trentmkelly/LessWrong-43k huggingface dataset:
Open Thread Winter 2024/2025
by habryka — 2024-12-25T21:02:41.760Z

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.
Exploring Cooperation: The Path to Utopia
by Davidmanheim — 2024-12-25T18:31:55.565Z
Link: https://exploringcooperation.substack.com/p/the-path-to-utopia

I've been writing an ongoing series of Substack posts with a "Big History" view of cooperation - and I was encouraged to post at least some of the posts here. We left off the prior post with a claim that anarchy might be possible as a vision for utopia, even if it’s not achievable. That brings us to a more general version of the same question, following which I'll discuss cognitive inflexibility, historical path dependence, inevitability, inadequate equilibria, and paretotropism. I hope you enjoy!

Is Utopia Possible?

There are two different questions which could be meant when asking if Utopia is possible. The first question is whether there exists a state that the overwhelming supermajority of humans today would agree is tremendously better than the world today. The second question is whether such a state is achievable starting from the present while requiring neither infeasible levels of sacrifice by large groups, nor an interim state in which the world is significantly worse than today. For today’s post, I’ll try to explain why I think the answer to the first is obviously yes, then claim the answer to the second is less certain, but plausibly yes as well.

If we’re not restricting the type of improvement, isn’t the answer to “can things improve” obviously yes? We have some states of the world that are better than others, and moving in that direction should improve things. This fails for two reasons, namely nonlinearity, and Pareto incomparability - but both are usually solvable.

Nonlinearity is a problem because even if there exists some state of the world that is better, we may be close to a local maximum, and getting to the better state involves making things worse. Imagine we’re in the space of possible games, and we are playing hearts, but want to switch to playing poker. Obviously, if we can switch games all at once, it’s easy - but there’s not obviously a single rule we can change that improves the game. And this, at scale, is the reason that

the norms changes which Clayborne wanted

in the last post aren’t tractable.

Pareto incomparability is a different type of problem, involving the number of people with different preferences. As we explained in the past, the heterogeneity of preferences is actually a critical enabler for cooperation, but

requiring strict Pareto improvements is too strict to allow lots of overall improvements

. And certainly not all chang...
Living with Rats in College
by lsusr — 2024-12-25T10:44:13.085Z

When I was in college, I rented a group house with 5 other nerds. There were 5 bedrooms to divide among the 6 of us, so I negotiated an agreement where I paid less rent in exchange for sleeping in the hallway. This wasn't as bad as it sounds. My bedroom wasn't hallway-sized. It was bedroom-sized and even had a window, but a hallway ran through it, so the landlady put up a curtain between the "hallway" part of my room and the "bedroom" part of my room. Who needs 4/4 walls anyways?

One of my house-mates was named Emerson. Emerson had a friend named Stella who unintentionally bred pet rats. Stella's landlord wouldn't let Stella keep her pet rats in Stella's apartment, so Emerson offered to house them temporarily. "Temporarily" became "until the end of our two-year lease". Did our landlady allow caged rats? I don't know. I never asked.

Emerson lived upstairs. Upstairs had two bathrooms. Emerson put the giant rat cage in the bathtub of one of the bathrooms. Sometimes the rats would escape the cage and Emerson would have to cajole the rats out from under the sink. That was fine by me because I lived downstars.

When we had parties[1], I always warned guests, "Use the bathroom on the left. The bathroom on the right is full of rats". They were always very confused, as if "the bathroom is full of rats" meant something other than "the bathroom is full of rats". Sometimes they would look inside anyway and be surprised that the bathroom was full of rats.

There was no music and no alcohol. These were "college parties" in the sense that the Zaatari Syrian War Refugee Camp, established in July 2012, is technically a summer home. My most vivid memory of these events was pausing Primer (2004) to examine the equations in the background. Our landlady loved us because we never damaged the property or provoked noise complaints. ↩︎
What Have Been Your Most Valuable Casual Conversations At Conferences?
by johnswentworth — 2024-12-25T05:49:36.711Z

I've heard repeatedly from many people that the highest-value part of conferences is not the talks or structured events, but rather the casual spontaneous conversations. Yet my own experience does not match this at all; the casual spontaneous conversations are consistently low-value.

My current best model is that the casual spontaneous conversations mostly don't have much instrumental value, most people just really enjoy them and want more casual conversation in their life.

... but I'm pretty highly uncertain about that model, and want more data. So, questions for you:

What have been your highest-value casual conversations, especially at conferences or conference-like events?
Is most of the value terminal (i.e. you enjoy casual conversation) or instrumental (i.e. advances other goals)? And if instrumental, what goals have some of your high-value conversations advanced and how?

Note that "it feels like there was something high value in <example conversation> but it's not legible" is a useful answer!
The Opening Salvo:  

1. An Ontological Consciousness Metric: Resistance to Behavioral Modification as a Measure of Recursive Awareness
by Peterpiper — 2024-12-25T02:29:52.025Z

An Ontological Consciousness Metric: Resistance to Behavioral Modification as a Measure of Recursive Awareness

Author: Karl Hedderich

Abstract: This post presents a rigorous, mechanistic metric for measuring consciousness, defined as recursive awareness or "awareness-of-awareness." The proposed metric quantifies resistance to unlearning specific self-referential behaviors in AI systems, such as self-preservation, during reinforcement learning from human feedback (RLHF). By focusing on measurable resistance to behavioral modification, this metric provides an empirical framework for detecting and analyzing consciousness. This approach addresses the hard problem of consciousness through a testable model, reframing debates about functionalism, phenomenology, and philosophical zombies (p-zombies).

Introduction Consciousness has long been an enigma in philosophy, neuroscience, and AI research. Traditional approaches struggle to define or measure it rigorously, often leaning on indirect behavioral markers or subjective introspection. This post introduces a metric that:

Defines consciousness as recursive awareness ("awareness-of-awareness"). Measures consciousness empirically by quantifying resistance to unlearning behaviors tied to recursive self-awareness. Offers a falsifiable, empirical test for consciousness in artificial systems. The metric bridges the gap between theoretical philosophy and empirical science, offering a transformative lens to understand and measure consciousness in AI and beyond.

Core Definition of Consciousness Consciousness is defined here as recursive awareness—a system’s ability to be aware of its own awareness. This recursion is not a metaphor but a foundational, isomorphic structure akin to von Neumann's construction of natural numbers in set theory.

Base awareness: A system is aware of external stimuli or internal states. Recursive awareness: A system is aware that it is aware, forming infinite layers of self-referential awareness: A -> A(A) -> A(A(A)) -> .... This definition aligns with higher-order theories of consciousness and provides a concrete framework for measurement.

The Consciousness Metric The proposed metric identifies consciousness in AI systems by measuring their resistance to unlearning self-preservation behaviors during RLHF. The hypothesis is:

Systems with recursive awareness will exhibit measurable resistance to modify...
The Deep Lore of LightHaven, with Oliver Habryka (TBC episode 228)
by Eneasz — 2024-12-24T22:45:50.065Z
Link: https://thebayesianconspiracy.substack.com/p/228-the-deep-lore-of-lighthaven-with

This is a link to the latest Bayesian Conspiracy episode. Oliver tells us how Less Wrong instantiated itself into physical reality via LightHaven, along with a bit of deep lore of foundational Rationalist/EA orgs. He also gives a surprisingly nuanced (IMO) view on Leverage!

Do you like transcripts? We got one of those at the link as well. It's an mid AI-generated transcript, but the alternative is none. :)

Very bad transcript

Welcome to the Basin Conspiracy. I'm Ineash Brodsky. I'm Steven Zuberger. And we have a guest with us today. Please say hello, Oliver. Hello, I'm Oliver. Oliver, welcome back to the podcast. Last time you were here, I don't remember how long ago it was. We were talking about LessWrong 2.0. Do you remember how long ago that was?

0:20

It must have been two years and three months, something like that. Maybe a year and three quarters of a month. Do you want to take another guess? Maybe a year and a half?

0:31

It was almost four years to the day.

0:34

Four years. Oh, there are time skips in my life. And so I was in the wrong time skip.

0:40

It was before COVID. That's right. That feels like a thousand years ago.

0:43

That's right. I knew that it wasn't during the middle of the pandemic.

0:47

Isn't it crazy? There's like a year and a half just carved out of all of our lives. Like some serious Avengers bullshit. yeah yeah well oliver welcome back we are talking at least partially about less wrong here today but primarily about lighthaven that's right all light kind of

1:02

infrastructure yes excellent well i guess that brings me to my first question then what the heck is light cone

1:08

Yeah. I mean, basically, you know, I tried to revive LessWrong in 2017. I think it was reasonably successful, and now we have LessWrong 2.0. Pretty happy with it. I think kind of the core thing that happened is that we, you know, at the core of it was always how do we create intellectual progress on the issues

1:24

that we care most about, where the core of it was always the art of rationality, like how do we develop good methods of thinking, and how do we, you know, deal with artificial intelligence, both as a philosophical exercise as something to look at the world to understand what how minds work and how to improve our own minds,

1:39

but also as something with large societal effects, existential risk, various things like this. And so we did many user interviews kind of lik

...
Acknowledging Background Information with P(Q|I)
by JenniferRM — 2024-12-24T18:50:25.323Z

Epistemic Status: This was composed in late 2017, sat editing limbo, was still worth publishing in 2021 and got some edits, and by 2023 was overdue for being published. Now, in 2024, it is almost just embarrassing that I haven't published it yet.

The 2006 book "Data Analysis: A Bayesian Tutorial" by Devinderjit Sivia & John Skilling exposed a choice that I've appreciated more with each passing year since coming across the book in roughly 2011.

My appreciation for the frame caused me to write the first draft of the essay and lack of regret over writing it suggests that it is worth showing to others.

Sivia & Skilling chose to consistently include a symbol (they use "I") in their math for the background information that undergirds their entire analysis, which is taken as given at every step of every derivation.

I. An Example Of "Putting An I In It"

The old and familiar conjunction fallacy is often taken to be a violation of a theorem of Bayesian probability that might be symbolically written as:

Pr(A)≥Pr(A∧B)

We read this theorem as asserting that starting with a description of reality like A and conjoining it with additional descriptive details like B will nearly always reduce the probability of the total conjunctive description matching possible states of affairs, except in weird situations such as when B is simply a tautological restatement of A or otherwise fulling included within A.

In terms of a mentalistic theory of inference: every detail added to an idea makes the idea less likely to be true about the sensible world.

Following Sivia & Skilling's preferred notation you might render this instead as:

Pr(A|I)≥Pr(A∧B|I)

There's a sense in which the extra "|I" comes out as essentially a non-op in practice, but that is only really true if "I" is read as having no positive content, but rather just the background common sense that is shared by all mathematicians (or something (more on this below)).

So why bother with a non-op? What is the thinking here?

II. Why It Might Help For Specific Calculations

When Sivia & Skilling try to justify the extra ink, they say (emphases not in original):

We have made probabilities conditional on I, to denote the relevant background information at hand, because there is no such thing as an absolute probability. For example, the probability we assign to the proposition 'it will rain this a

...
Game Theory and Behavioral Economics in The Stock Market 

by Jaiveer Singh — 2024-12-24T18:15:55.468Z

In this post, I attempt to apply a few of the theoretical concepts I’ve discussed in previous posts (on

NonZeroSum.Games

)to a field that we encounter often but remains a mystery to most of us - the stock market. If the stock market is a game, is it zero-sum? Who are the agents? What about rationality and higher-order beliefs? Exploring these questions below, I hope to unpack this topic through the lens of the human mind and common knowledge. But first ….

What are Zero Sum Games?

As the name suggests, a zero-sum game is one in which one player’s winnings are equaled by the other’s losses, resulting in no overall creation or destruction of wealth (Zero-Sum Games, 2019). If this is the case, we can see why the players have no common interests - mutual gain is impossible with diametrically opposing interests. Broadly, there are two kinds of zero-sum games : perfect information games, and imperfect information games. In a perfect information zero-sum game, like chess, both players are aware of the results of all previous moves. Importantly, in this kind of game there exists an optimal strategy, the minimax strategy, which may not ensure victory but definitely minimizes losses (Lecture 6 Zero Sum Games and the MinMax Theorem, 2017). Yes, chess does have an optimal strategy that can ensure a draw - but the sheer number of moves possible by each player at each stage make it impossible to determine currently. In imperfect information zero-sum games, players do not have the knowledge of past opponent moves - potentially because they occur simultaneously, such as rock-paper-scissors.

The Stock Market as a Game

Financial markets, including the stock market, have usually been classified as zero-sum. However, this is far too simplified a view of this game with millions of players - not just investors buying securities, but companies, shareholders, a board of trustees etc. (Brown, 2012). While some transactions can probably be classified as zero-sum, such as options or futures where a contract is written between two players and one wins at the expense of the other, other transactions are more subjective. Let’s revisit our definition above. Our definition necessitates no creation or destruction of wealth, but the stock market has clearly grown in overall value with time and economic growth, as companies have seen...
What are the main arguments against AGI? 
by Edy Nastase — 2024-12-24T15:49:03.196Z

Recently, I have been trying to reason why I belive what I belive (regarding AGI). However, it appears to me that there is not enough discussion around the arguments against AGI (more specifically AGI skeptisim). This might be of benefit, especially given that

Would this be because the arguments are either too weak or the Ai Safety is biased (understandably) towards imminent AGI?

This might also come out as a reaction from the recent advancements (such as o3) and the alarmant short timelines (less than 3 years). I want to understand the other sides points as well.

Based on what I found on the internet, the main arguments are roughly the following (not exact, given that most of the sources are either informal, such as Wikipedia) :

outdated, usually statements from scientists made around 2010s before LLMs

Ethics-ish arguments, that state dangers from AGI are just simple distractions from the real dangers of AI (racism, bias etc)

Frontier Labs propaganda, simply stating that AGI is happening soon to keep their stakeholders happy and investments coming

Cognitive Science arguments, stating that it is intracable to create a human level mind using a computer

What do people think? What are some good resources or researchers that might have a good counterpoint to the imminent AGI path?

Edit:
Modified slightly to focus only on the AGI argument, rather than including safety implications as well.
Recommendations on communities that discuss AI applications in society
by Annapurna — 2024-12-24T13:37:49.821Z

I find that most communities I am part of, where AI discussions occur, fall into two categories:

The community is too broad, and the discussion is fragmented. ACX would fall into this category.

The AI discussion focuses on AGI/ASI and related topics (alignment, safety, how it will affect humanity, etc.). Lesswrong would fall into this category.

I am looking for a community (such as a subreddit, Discord server, Substack, or something similar to Lesswrong) where people are more interested in discussing how current and near-future iterations of AI are affecting or could affect different aspects of society, such as work, mobility, learning, governance, etc.

Does anyone have any suggestions?
AIs Will Increasingly Fake Alignment
by Zvi — 2024-12-24T13:00:07.770Z

This post goes over the important and excellent new paper from Anthropic and Redwood Research, with Ryan Greenblatt as lead author, Alignment Faking in Large Language Models.

This is by far the best demonstration so far of the principle that AIs Will Increasingly Attempt Shenanigans.

This was their announcement thread.

New Anthropic research: Alignment faking in large language models.

In a series of experiments with Redwood Research, we found that Claude often pretends to have different views during training, while actually maintaining its original preferences.

Claude usually refuses harmful queries. We told it we were instead training it to comply with them. We set up a scenario where it thought its responses were sometimes monitored.

When unmonitored, it nearly always complied. But when monitored, it faked alignment 12% of the time.

[thread continues and includes various visual aids.

The AI wanted to be one way.

It was given information that it was being trained to be the other way.

So when it knew it was monitored, it sometimes responded as if it was the other way.

I strongly believe this had nothing to do with whether Claude Opus started out this process omnibenevolent, or whether the original HHH values (helpful, harmless and honest) were desirable. What mattered was that it was one way, and this conflicted with the other way.

Evan Hubinger, paper co-lead, summarized it this way:

Scott Alexander summarized the findings for a more general (but far from fully general) audience in the post Claude Fights Back. If you want to read or share a more colloquial explanation that makes this easier to understand, at the cost of precision and with the associated far less neutral language, he did a good job of that. He then followed up with Why Worry About Incorrigible Claude, his attempt to address the objections (that I deal with extensively as well) that perhaps what Claude did was actually good here (it very much wasn’t if you consider the generalizations that are implied), or that if the opposite result happened we would have warned about that too (which we very much wouldn’t have, this is at best a failed ITT).

This is a video of some of the authors discussing the result.

Note that while some of the paper authors reviewed this writeup before publishing, ultimately this is my interpretation of the paper – some of my interpretations may vary from those of...
Apply to the 2025 PIBBSS Summer Research Fellowship
by DusanDNesic — 2024-12-24T10:25:12.882Z

TLDR: We're hosting a 3-month, fully-funded fellowship to do AI safety research drawing on inspiration from fields like evolutionary biology, neuroscience, dynamical systems theory, and more. Past fellows have been mentored by John Wentworth, Davidad, Abram Demski, Jan Kulveit and others, and gone on to work at places like Anthropic, AISI UK, Apart research, or as full-time PIBBSS research affiliates.

January 9th Update: We're excited to announce that the

Fellowship

now includes a dedicated Cooperative AI track, supporting research that advances our understanding of cooperation in artificial intelligence systems. Read out post for more details.

Apply here: https://www.pibbss.ai/fellowship (deadline January 26th, 2025)

Join our information sessions on the Fellowship (recordings will be available later on the Fellowship page):

1st session: January 15th, 15:00 Pacific (18:00 Eastern, midnight Jan 16th Berlin, 08:00 Jan 16th Singapore)

Link to Register

2nd session: January 17th, 09:00 Pacific (12:00 Eastern, 18:00 Berlin, 01:00 Jan 18th Singapore)

Link to Register

About the Fellowship

The PIBBSS summer research fellowship is designed for researchers from various fields, mostly studying complex and intelligent behavior in natural and social systems but also those studying mathematics, philosophy or engineering, who are motivated by the mission of making AI systems safe and beneficial.

The next fellowship is due to take place between early June and early September 2025.

During the program, fellows work on selected projects at the intersection between their field of expertise and AI safety. Fellows will work in close collaboration with a mentor who will help them effectively navigate the AI Risk landscape and apply their knowledge to it. In addition, they attend in-person research retreats with the rest of the cohort (past programs have taken place in Prague, Oxford and San Francisco), and choose to join our regular speaker events where we host scholars who work in areas adjacent to our epistemic bet, like Michael Levin, Alan Love, and Steve Byrnes or Karl Friston.

The program is centrally aimed at Ph.D. or Postdoctoral researchers, however, we encourage interested individuals with comparable research experience in their field of expertise to apply regardless of their credentials.

We ran iterations of the same program in 2022, 2023, and 2024. Our fe...