trentmkelly/LessWrong-43k huggingface dataset:
Motivated Cognition and the Multiverse of Truth
by Q Home — 2022-11-22T12:51:26.405Z

Related to: Truth seeking is motivated cognition and Contingency is not arbitrary, Do Sufficiently Advanced Agents Use Logic?

I have a weird belief, it's both controversial and obscure:

Motivated Cognition (MC) has a serious chance to be a good epistemology.

Even if MC is wrong, it's very useful for goals which are not about predicting reality. And MC is a path to ideas which don't require MC, but which are hard to introduce without it.

Properly knowing the idea of MC and knowing how to apply it makes any person ~2 times smarter, more perceptive and understanding.

There is an unexpected duality between Bayesian reasoning and MC. I'm not confident in this part of my belief, but I think this should be true at least to some degree.

I want to have a rational and semi-rational discussion about MC. "Semi-rational" meaning you're ready to suspend some judgement to fully appreciate the idea. If this idea is true it's as important as the Scientific method or the Bayes' theorem, so the stakes are very high. The post contains all cruxes of my belief, so I'm ready for a double-crux. Sorry if I write too arrogantly in a couple of places in the post, I just want to emphasize my point.

I don't want to "shock" people with my belief. I know, it doesn't sound nice, but it's original and doesn't hurt to entertain. And I realize that it also sounds too ambitious: what's that part about "2 times smarter", how can I know that and what does that even mean? I hope my post will be able to clarify this.

I believe that at the very least analyzing MC could help us understand people who criticize and disagree with rationalist community. I think this is by itself is important enough.

Intro: what is Motivated Cognition?

Two types of MC

Let's differentiate two types of motivated cognition:

Local wishful thinking is about random wishes that don't take into account facts. "I want to fly, so I jump off of a cliff towards injury and death."

Global motivated cognition (MC) is about your most important wish + facts.

Logical thinking is about "logical inferences + facts", MC is about "wishes + facts".

You can't use logic without all facts and all inferences. The same way you shouldn't use MC without all facts and all wishes.

How can MC possibly work?

Note that typical criticisms of MC may be unfair:

Reality doesn't care about your wishes.

Reality doesn't care about any thinking method.

You ca

...
LessWrong readers are invited to apply to the Lurkshop
by Jonas V — 2022-11-22T09:19:05.412Z

Summary: We’re running a free retreat for LessWrong users (lurkers) who aren’t yet working on existential risk mitigation but might be interested, in the hopes of catalyzing some of them into people who are. It’s called the Lurkshop.

Apply here

by December 1st, ideally sooner! If you refer a friend and we accept them, we’ll give you $100. We cover all costs, including flights. Please apply even if you don't think you have skills that seem directly relevant to reducing x-risk!

--

Do you read LessWrong?

(Presumably!)

Are you doing anything about the dire existential warnings you're hearing about?

(Probably not!)

If you want to be, we're offering an all-expenses paid workshop, Dec 16 (noon) to Dec 18 (10pm) in Berkeley, CA to come meet community members, talk with each other, and otherwise engage with the main plot of the world. If this goes well we might offer more programs like this, so please apply even if the dates don’t work well for you and we might invite you to those future programs!

What? Why?

LessWrong is one of the places in the world where words mean things and people think openly about the hardest problems facing the world, and we'd like to get to know the people who read it a bit better and help them make progress on the world’s problems.
The rationality community mostly exists on online forums. Forums are great, but it’s really hard to replicate some important aspects of in-person conversations.

You can have different kinds of interactions and conversations in face-to-face interactions, which are useful for building trust, context, and deeper relationships

It’s helpful to be able to have conversations that aren't mostly in public. There's a lot that people want to talk about that's sensitive or personal or private
It’s helpful to be able to have long conversations with lots of context that are laser focused on you and your situation
You probably can't find a cofounder just by writing LessWrong comments.

Forums make the community feel remote and actions feel impossible

The rationality community is not some distant mythical group—it’s a bunch of people!
Many people quite like you are doing big ambitious risky projects, and seeing that helps people realize they can do those kinds of projects too
There really isn’t someone on top of all the important stuff. Come see for yourself how disorganized and incompetent everyone is!

Notably, this retreat probab...
Gaoxing Guy
by Alok Singh — 2022-11-22T01:50:07.666Z

August 2016: Jiuzhaigou, Sichuan Province, China[^maybe it should be timespace instead of spacetime, better to put the time first, what if the place identifier no longer exists? Plus timestamps are orderable.]

Cab driver in front, me in back of cab. Awful traffic from Jiuzhaigou, after a long day of the most physically dangerous thing I’ve ever done. Two parallel lines of cars extended, to the point where parallels meet.

he: 好的? (Good?)

me: 我高兴九寨沟

he: 不对! (wrong)

Eyebrows arch.

me: 什么 (what?)

(lots of his finger pointing)

he: 我喜欢他, 我喜欢这个, 他喜欢呢个 (I like him, I like this, he likes this)

Ah.

me: 我喜欢九寨沟 (I like Jiuzhaigou)

He smiles.

Throughout the conversation, a realization percolated within.

Language \(≠\) communication (and that I implicitly identified them)

This guy was communicating better with snippets and his hands than me and my constricting words. Dropping all that and just conversing freely without those constraints reminded me of what naivete can provide.
Miscellaneous First-Pass Alignment Thoughts
by NickGabs — 2022-11-21T21:23:53.960Z

I have been learning more about alignment theory in the last couple of months, and have heard from many people that writing down naive hypotheses can be a good strategy for developing your thoughts and getting feedback about them. So here goes:

I am skeptical that any training mechanism that we invent will so consistently avoid inner misalignment that we can use it to successfully align an AGI without supervision and rollback if the model we are training begins to become misaligned. As a result, good interpretability tools seem like a necessary part of almost all alignment schemes because they are necessary for doing this kind of oversight well; it makes sense that they feature heavily in Evan Hubinger's post on different alignment proposals.

Conversely, sufficiently good interpretability tools seem close to sufficient for solving alignment. This is because using them, we could prevent inner misalignment with a high degree of probability, and because they are likely to make other parts of alignment easier by allowing us to understand neural networks better in general. Also, they are the main component of John Wentworth's "Retarget the Search" strategy, which, assuming its assumptions hold (that human values are a sufficiently natural abstraction that we will be able to locate them in a sufficiently intelligent world modeling AI), could plausibly solve alignment.
Moreover, I buy the arguments in this post that we can plausibly make a lot of progress on interpretability (tl;dr we have made lots of progress on neuroscience, which is basically interpretability for the human brain, and in a lot of ways NN interpretability seems easier)

In addition, I also think something like the natural abstraction hypothesis is true. In particular, it seems implausible that there is another way to abstract about the macroscopic physical world which is nearly as good as the way humans understand it (i. e. a set of 3D objects [and roughly the same set which we carve the world into] interacting in Newtonian-ish ways). Similarly, our understanding of people as agents that act to accomplish goals as opposed to "mechanistic systems" seems like an extremely powerful abstraction without another that is equivalently good for reasoning about the behavior of people. Thus, I expect AI's to have similar abstraction to us in at least two significant do

...
[Hebbian Natural Abstractions] Introduction
by Samuel Nellessen — 2022-11-21T20:34:15.298Z
Link: https://www.snellessen.com/introduction-to-hebbian-natural-abstractions

With this sequence, we (Sam + Jan) want to provide a principled derivation of the natural abstractions hypothesis (which we will introduce in-depth in later posts) by motivating it with insights from computational neuroscience.

Goals for this sequence are:

show why we expect natural abstractions to emerge in biological brains,
provide empirical evidence and a mechanistic explanation for the emergence of natural abstractions in biology, and
spelling out implications of the emergence of natural abstractions in biology.

Author’s note: This is currently my (Sam’s) main research project, but my first nonetheless. Happy to receive any feedback! Some of the original ideas and guidance come from Jan. I don’t expect you, the reader, to have solid background knowledge in any of the discussed topics. So, whenever you get lost, I will try to get you back on board by providing a more high-level summary of what I said.

Why are ‘abstractions’ relevant?

As alignment researchers, the higher-level problem we are trying to solve is: ‘How do we teach an AI what we value?’. To simplify the question, we assume that we already know what we value[1]. Now, we’ve got to teach an AI that we value “things out there in the world”, e.g. trees. Specifying “trees” should be easy, right?

… o no …

You: Yea, so… we value trees. These for example. gestures wildly
AGI: Of course! So you only value these trees in your backyard? Okay, I will just quickly measure the constellation of molecules in—
You: No, no, no! Stop, that’s not what I meant! Not only those, but you know… the idea behind the trees? I also value trees that aren’t present here and certainly not their constituting molecules. Look, let’s go over to my neighbors backyard. These are trees too.
AGI: Huh, this tree seems to be made out of papier-mâché.
You: Ahhh uups… Yea, humans are fallible. Don’t treat my best guess estimations as unfailing sources of truth. Please note that I care about the territory - not just the map in my head that might be mistaken.
AGI: Wh- What? How should I know about the relation between your map and the territory?

In a similar vein, John Wentworth spells out the pointers problem[2]: ‘An AI should optimize for the real-world things I value, not just my estimates of those things’. He formalises the problem as follows:

“What functions of what variables (if any) in the environment and/or another w

...
Utilitarianism Meets Egalitarianism
by Scott Garrabrant — 2022-11-21T19:00:12.168Z

This post is mostly propaganda for the Nash Bargaining solution, but also sets up some useful philosophical orientation. This post is also the first post in my geometric rationality sequence.

Utilitarianism

Let's pretend that you are a utilitarian. You want to satisfy everyone's goals, and so you go behind the veil of ignorance. You forget who you are. Now, you could be anybody. You now want to maximize expected expected utility. The outer (first) expectation is over your uncertainty about who you are. The inner (second) expectation is over your uncertainty about the world, as well as any probabilities that comes from you choosing to include randomness in your action.

There is a problem. Actually, there are two problems, but they disguise themselves as one problem. The first problem is that it is not clear where you should get your distribution over your identity from. It does not make sense to just take the uniform distribution; there are many people you can be, and they exist to different extents, especially if you include potential future people whose existences are uncertain.

The second problem is that interpersonal utility comparisons don't make sense. Utility functions are not a real thing. Instead, there are preferences over uncertain worlds. If a person's preferences satisfy the VNM axioms, then we can treat that person as having a utility function, but the real thing is more like their preference ordering. When we get utility functions this way, they are only defined up to affine transformation. If you add a constant to a utility function, or multiply a utility function by a positive constant, you get the same preferences. Before you can talk about maximizing the expectation over your uncertainty about who you are, you need to put all the different possible utility functions into comparable units. This involves making a two dimensional choice. You have to choose a zero point for each person, together with a scaling factor for how much their utility goes up as their preferences are satisfied.

Luckily, to implement the procedure of maximizing expected expected utility, you don't actually need to know the zero points, since these only shift expected expected utility by a constant. You do, however need to know the scaling factors. This is not an easy task. You cannot just say something like "Make all the scaling factors 1." You don't actually start with util...
Interview with Matt Freeman
by  — 2022-11-21T18:17:43.330Z
Link: https://overcast.fm/+8wMCpdpkI

An interview with Matt Freeman, co-founder of the Guild of the ROSE, an organization for rationalist community and education. Topics discussed include:

The motivation and vision behind creating the Guild.
What might be possible with consistent rationality training inside a community of rationalists?
The importance of understanding practical decision theory.
Here's the exit.
by Valentine — 2022-11-21T18:07:23.607Z

There's a kind of game here on Less Wrong.

It's the kind of game that's a little rude to point out. Part of how it works is by not being named.

Or rather, attempts to name it get dissected so everyone can agree to continue ignoring the fact that it's a game.

So I'm going to do the rude thing. But I mean to do so gently. It's not my intention to end the game. I really do respect the right for folk to keep playing it if they want.

Instead I want to offer an exit to those who would really, really like one.

I know I really super would have liked that back in 2015 & 2016. That was the peak of my hell in rationalist circles.

I'm watching the game intensify this year. Folk have been talking about this a lot. How there's a ton more talk of AI here, and a stronger tone of doom.

I bet this is just too intense for some folk. It was for me when I was playing. I just didn't know how to stop. I kind of had to break down in order to stop. All the way to a brush with severe depression and suicide.

And it also ate parts of my life I dearly, dearly wish I could get back.

So, in case this is audible and precious to some of you, I'd like to point a way to ease.

The Apocalypse Game

The upshot is this:

You have to live in a kind of mental illusion to be in terror of the end of the world.

Illusions don't look on the inside like illusions. They look like how things really are.

Part of how this one does the "daughter's arm" thing is by redirecting attention to facts and arguments.

"Here's why the argument about AI makes sense."
"Do you have some alternative view of what will happen? How do you address XYZ?"
"What makes it an 'illusion'? I challenge that framing because it dismisses our ability to analyze and understand yada yada."

None of this is relevant.

I'm pointing at something that comes before these thoughts. The thing that fuels the fixation on the worldview.

I also bet this is the thing that occasionally drives some people in this space psychotic, depressed, or into burnout.

The basic engine is:

There's a kind of underlying body-level pain. I would tag this as "emotional pain" but it's important to understand that I really am pointing at physical sensations.
The pain is kind of stored and ignored. Often it arose from a very young age but was too overwhelming, so child-you found methods of distraction.
This is the basic core of addiction. Addictions are when there's an intolerable sensatio

...
Benefits/Risks of Scott Aaronson's Orthodox/Reform Framing for AI Alignment
by Jeremyy — 2022-11-21T17:54:01.401Z
Link: https://scottaaronson.blog/?p=6821

Scott Aaronson makes the argument that an Orthodox vs Reform analogy works for AI Alignment. I don't think I've really heard it framed this way before. He gives his takes on the beliefs of each group and there's a lot of discussion in the comments around that, but what's more interesting to me is thinking about what the benefits and risks of framing it this way might be.

Perhaps the reform option gives people a way to take the arguments seriously without feeling like they are aligning themselves with something too radical. If your beliefs tend reformist, you probably like differentiating yourself from those with more radical-sounding beliefs. If your beliefs are more on the Orthadox side, maybe this is the "gateway drug" and more talent would find its way to your side. This has a little bit of the "bait-and-switch" dynamic I sometimes hear people complain about (but I do not at all endorse) with EA - that it pitches people on global health and animal welfare, but it's all really about AI safety. As long as people really do hold beliefs along reformist lines though, I don't see how that would be an issue.

Maybe the labels are just too sloppy, most people don't really fit into either camp and it's bad to pretend that they do?

Not coming up with much else, but I'd be surprised if I wasn't missing something.
[ASoT] Reflectivity in Narrow AI
by Ulisse Mini — 2022-11-21T00:51:39.168Z

I wrote this a month ago while working on my SERI MATS applications for shard theory. I'm now less confident in the claims and the usefulness of this direction, but it still seems worth sharing.

I think reflectivity happens earlier then you might think in embedded RL agents. The basic concepts around value drift ("addiction", ...) are available in the world model from pretraining on human data (and alignment posts), and modeling context dependent shard activation and value drift helps the SSL WM predict future behavior. Because of these things I think we can get useful reflectivity and study it in sub-dangerous AI. This is where a good chunk of my alignment optimism comes from. (understanding reflectivity and instrumental convergence in real systems seems very important to building a safe AGI.)

In my model people view reflectivity through some sort of magic lens (possibly due to conflating mystical consciousness and non-mystical self-awareness?). Predicting that I'm more likely to crave cookie after seeing cookie or after becoming hungry isn't that hard. And if you explain shards and contextual activation (granted understanding the abstract concepts might be hard) you get more. Abstract reflection seems hard but still doable in sub AGI systems.

There's also a meaningful difference between primitive/dumb shard, my health shards outmaneuver my cookie shards because the former are "sophisitcated enough" to reflectively outsmart the cookie shards (e.g. by hiding the cookies) while the latter is contextually activated and "more primitive". I expect modelling the contextual activation of the cookie shard not to be hard, but reflective planning like hiding cookies seems harder- though still doable. (This might be where the majority of the interesting/hard part could be.)
Scott Aaronson on "Reform AI Alignment"
by Shmi — 2022-11-20T22:20:23.895Z
Link: https://scottaaronson.blog/?p=6821

The framing is based on contrasting Orthodox Judaism with Reform Judaism, only for AI Alignment.

If AI alignment is a religion, it’s now large and established enough to have a thriving “Reform” branch, in addition to the original “Orthodox” branch epitomized by Eliezer Yudkowsky and MIRI. As far as I can tell, this Reform branch now counts among its members a large fraction of the AI safety researchers now working in academia and industry. (I’ll leave the formation of a Conservative branch of AI alignment, which reacts against the Reform branch by moving slightly back in the direction of the Orthodox branch, as a problem for the future — to say nothing of Reconstructionist or Marxist branches.)

There are 8 points of comparison, here is just the first one:

(1) Orthodox AI-riskers tend to believe that humanity will survive or be destroyed based on the actions of a few elite engineers over the next decade or two. Everything else—climate change, droughts, the future of US democracy, war over Ukraine and maybe Taiwan—fades into insignificance except insofar as it affects those engineers.

We Reform AI-riskers, by contrast, believe that AI might well pose civilizational risks in the coming century, but so does all the other stuff, and it’s all tied together. An invasion of Taiwan might change which world power gets access to TSMC GPUs. Almost everything affects which entities pursue the AI scaling frontier and whether they’re cooperating or competing to be first.

This framing seems new and worthwhile, crystallizing some long-standing divisions in the AI community.
On Morality, Ethics, and all that Jazz
by Delen Heisman — 2022-11-20T20:00:42.354Z
Link: https://delen.substack.com/p/on-morality-ethics-and-all-that-jazz

Let’s start with some definitions first to make sure that we are all on the same page. I have no idea what the formal definitions are in this space but hopefully this will be enough for mutual understanding.

Agent: Anything that can (seemingly) act to affect the universe around them. Disregard questions of determinism/free-will as they relate here but shouldn’t matter in this context.

Meta-Agent: An agent that is self-aware. In other words: an agent that recognizes its own agent-hood.

State: A configuration of the universe at a given time.

Most Preferred State (MPS): A state in which they would not choose (rank higher) any other state.

Least Preferred State (LPS): A state in which they would choose (rank higher) any other state.

My claims:

Humans are meta-agents.
Other meta-agents can (do?) exist.
Every meta-agent will have a state graph over time that shows for any time (t) there is a state (S) somewhere in between their MPS and LPS.
Maximization of the function S(t) is the thing that meta-agents attempt to do by their actions. One approach would be something like aiming for maximal area under the curve along with a discount factor as time gets more distant from the present. However, it would seem to me there could be many different strategies here.
While it is possible to have ethical/moral frameworks that only consider meta-agents that are humans it is more useful[1] to talk about frameworks that consider all meta-agents (animals, aliens, AI, etc).
Any system of ethics/morals (that applies to all meta-agents) should provide a framework for meta-agents to accomplish (4) and therefore should be judged on their ability to do that.

OK this is great and all but why should we care?

I’ve never seen anyone lay out this sort of framework. Someone may have done it but if they have it certainly is not well known. And consequently I find that when you view ethical/moral questions through this framework a lot of those questions become either trivial to answer or meaningless. For example let’s take a look at the definition of Moral Universalism per Wikipedia:

Moral universalism (also called moral objectivism) is the meta-ethical position that some system of ethics, or a universal ethic, applies universally, that is, for "all similarly situated individuals",[1] regardless of culture, race, sex, religion, nationality, sexual orientation, gender identity, or any other d

...