trentmkelly/LessWrong-43k huggingface dataset:
Rebuttals for ~all criticisms of AIXI
by Cole Wyeth — 2025-01-07T17:41:10.557Z

Written as part of the AIXI agent foundations sequence, underlying research supported by the LTFF.

Epistemic status: In order to construct a centralized defense of AIXI I have given some criticisms less consideration here than they merit. Many arguments will be (or already are) expanded on in greater depth throughout the sequence. In hindsight, I think it may have been better to explore each objection in its own post and then write this post as a summary/centralized reference, rather than writing it in the middle of that process. Some of my takes have already become more nuanced. This should be treated as a living document.

With the possible exception of the learning-theoretic agenda, most major approaches to agent foundations research construct their own paradigm and mathematical tools which are not based on AIXI. Nothing in 2024's shallow review of technical AI safety seems to advance the theory of AIXI or even use its tools. Academic publications on the topic are also quite sparse (in my opinion some of the last major progress took place during Jan Leike's PhD thesis in the 2010's). The standard justification is that AIXI fails as a realistic model of superintelligence, particularly the aspects we care about for alignment such as recursive self-improvement or the construction of superintelligence. Some of the criticisms have merit, but most of them are so informal that it is not clear whether or precisely why they are correct. The "why" is centrally important because if AIXI is an excellent model for a restricted setting of superintelligence (arguably far more appropriate than any competing model), as even its critics usually accept, and if it can be shown to fall short in some cases relevant to alignment obstacles, the precise details are likely to be useful for understanding the nature of those alignment obstacles, possibly even illuminating when those obstacles arise. Similarly, failures of AIXI as a recipe for constructing recursively self-improving superintelligence should illustrate how recursively self-improving superintelligence can and cannot arise. For these reasons, the natural response to encountering limitations of AIXI is to dig deeply into the mathematical details of those limitations and beyond. In fact, this is in line with the way that professional mathematicians and theoretical computer scientists would usually respond to analogous situat...
OpenAI #10: Reflections
by Zvi — 2025-01-07T17:00:07.348Z

This week, Altman offers a post called Reflections, and he has an interview in Bloomberg. There’s a bunch of good and interesting answers in the interview about past events that I won’t mention or have to condense a lot here, such as his going over his calendar and all the meetings he constantly has, so consider reading the whole thing.

Table of Contents

The Battle of the Board.

Altman Lashes Out.

Inconsistently Candid.

On Various People Leaving OpenAI.

The Pitch.

Great Expectations.

Accusations of Fake News.

OpenAI’s Vision Would Pose an Existential Risk To Humanity.

The Battle of the Board

Here is what he says about the Battle of the Board in Reflections:

Sam Altman: A little over a year ago, on one particular Friday, the main thing that had gone wrong that day was that I got fired by surprise on a video call, and then right after we hung up the board published a blog post about it. I was in a hotel room in Las Vegas. It felt, to a degree that is almost impossible to explain, like a dream gone wrong.

Getting fired in public with no warning kicked off a really crazy few hours, and a pretty crazy few days. The “fog of war” was the strangest part. None of us were able to get satisfactory answers about what had happened, or why.

The whole event was, in my opinion, a big failure of governance by well-meaning people, myself included. Looking back, I certainly wish I had done things differently, and I’d like to believe I’m a better, more thoughtful leader today than I was a year ago.

I also learned the importance of a board with diverse viewpoints and broad experience in managing a complex set of challenges. Good governance requires a lot of trust and credibility. I appreciate the way so many people worked together to build a stronger system of governance for OpenAI that enables us to pursue our mission of ensuring that AGI benefits all of humanity.

My biggest takeaway is how much I have to be thankful for and how many people I owe gratitude towards: to everyone who works at OpenAI and has chosen to spend their time and effort going after this dream, to friends who helped us get through the crisis moments, to our partners and customers who supported us and entrusted us to enable their success, and to the people in my life who showed me how much they cared.

We all got back to the work in a more cohesive and positive way and I’m very proud of our focus

...
Some implications of radical empathy
by MichaelStJules — 2025-01-07T16:10:16.755Z

Summary

In my first piece, I settled on conceiving of radical empathy as an object view. I highlight some potential moral implications of object views:

More of what someone finds pleasant or is (knowingly or unknowingly) disposed to find pleasant is better for them, all else equal. Death can be bad for them if and because it deprives them of the objects of those preferences. Death can also be bad for them if and because they explicitly disprefer it or its consequences (more).
We should care about what animals care about, e.g. preventing their pain and keeping them close to those to which they’re attached (more).
Someone’s suffering might not necessarily always count against their life, in case it isn't about their life or aspects of it, e.g. grieving the loss of a loved one (more).
If there can be (chemically-induced) pleasures and displeasures that aren’t about anything at all, they would not guide action on their own, although attitudes about them could (more).
The concept of ‘welfare’ may need to be substantially revised or even abandoned as a foundational concept, given its common characterization as self-regarding and only affected during the subject’s life by events during their life (more).

Acknowledgements

Thanks to Ariel Simnegar, Lukas Gloor, Justis Mills, Tori, JackM and Vasco Grilo for helpful feedback. All errors are my own.

The harm of death

It is worth remarking that the above asymmetry doesn’t mean it can’t be better to Alice to experience more pleasure or other positive attitudes or pro-attitudes in her life than otherwise if she will exist anyway. She may herself explicitly prefer to experience pleasure or other positive attitudes.

Or, perhaps, if she is disposed to enjoy skiing, say, then it’s better for her to ski more. She therefore has dispositional preferences for the things that would bring her pleasure, even things she doesn’t know or expect to enjoy. And similarly for other pro-attitudes. She can discover these unknown dispositional preferences of hers through experience.

In this case, it’s not (necessarily) the missing pleasure or other pro-attitudes themselves that make death bad, it’s just the lesser achievement of the objects of our dispositional pro-attitudes, to have less of the things we are disposed to have positive attitudes towards. This could ground a deprivationist&n...
Actualism, asymmetry and extinction
by MichaelStJules — 2025-01-07T16:02:31.610Z

Summary

I illustrate and defend actualist object views as my conception of radical empathy, of being concerned exactly with what we would actually care about. As a kind of asymmetric person-affecting view, the most important implication for cause prioritization is probably lower priority for extinction risk reduction relative to total utilitarianism.

I illustrate with an example where I find actualist object views handle changing preferences better than other views (more).
An important implication for cause prioritization is the (Procreation) Asymmetry: we have reasons to prevent miserable lives, but not to create happy or fulfilling lives, for the sake of what those lives would actually care about (more).

This is not (necessarily) antinatalist or pro-extinction, but it would probably lead to less priority for extinction risk reduction, compared to total utilitarianism (more).

I highlight some problems for actualism, addressing one dilemma here, and leaving issues with the Repugnant Conclusion and replacement for another piece (more).

Acknowledgements

Thanks to Ariel Simnegar, Lukas Gloor, Justis Mills, Tori, JackM and Vasco Grilo for helpful feedback. All errors are my own.

An example for changing preferences

There’s an adult named Alice and a child named Bobby. In the outcome Adopt, Bobby’s parents are killed and Alice meets and adopts Bobby, coming to care deeply about his welfare — her concern for his welfare is the preference P —, but she only met and adopted Bobby because his parents were killed in Adopt. In Adopt, their deaths have caused him some trauma, and he’d have been better off if they hadn’t been killed, in the outcome Survive. Still, Alice would be an excellent adoptive mother, and Bobby’s life would still seem very good overall to her in the world in which his parents are killed and she adopted him (Adopt), so she’d have a favourable or positive attitude towards his life and welfare (in Adopt). However, Alice, in Adopt, would be genuinely concerned for Bobby’s welfare and what’s best for him, and would know he’d have been better off had his parents not been killed and prefers that they wouldn’t have been killed. In Adopt, she’d therefore prefer Survive. Her attitude matches her impression of Bobby’s overall welfare.

Survive: Bobby’s parents survive, Alice never meets Bobby	Adopt: Bobby’s pa

...
Meditation insights as phase shifts in your self-model
by Jonas Hallgren — 2025-01-07T10:09:35.854Z

Introduction

In his exploration of "Intuitive self-models" and PNSE (Persistent Non-Symbolic Experience), Steven Byrnes offers valuable insights into how meditation affects our sense of self. While I agree with his core framework, I believe we can push this analysis further by examining how meditation fundamentally changes our model of personal boundaries and agency.

I propose that what we call "awakening" or "enlightenment" can be understood through the lens of predictive processing and information boundaries. Specifically, I suggest that meditation systematically alters our generative model of "self" - not just by removing the homunculus as Byrnes suggests, but by transforming how we perceive the <<Boundary>> between self and world.

To build this case, we need to start with a precise understanding of boundaries and agency. When we talk about "self," we're really talking about what we believe we have causal control over - our sphere of influence in the world. This boundary isn't fixed; it expands and contracts based on our experiences and beliefs.

In technical terms, we can model this using active inference and Markov blankets (different from the critch definition) - mathematical tools that help us understand how systems maintain separation from their environment while remaining causally connected. But rather than dive into the mathematics, let me offer a simple example that captures the essence of how these boundaries work in practice

Example:

Boundary around a human

The example that I like to give for what the effects of self-modelling through this frame is what happens with a water bottle, before and after you buy it. What is it that makes a water bottle yours or part of your extended boundary?

1. The water bottle is in the store:

You haven’t bought the water bottle and so it is just standing on that shelf. Someone else might come and pick it up, so if you have no plans of buying it, you have no causal control over the water bottle and so you can’t plan with it in mind and so it isn’t “your” water bottle.

2. You’ve bought the water bottle:

You’ve bought the water bottle and taken it home. It is now mainly affected by what variables your existing “self” does to it and so it extends under your control. It is now therefore an extended part of you.

The core question for your generative model is: “Is this under my causal control as defined by my existin...
Alleviating shrimp pain is immoral.
by G Wood — 2025-01-07T07:28:49.432Z

I read this article, felt emotional disgust at the argument and wondered why? I don't really want to hurt shrimp, so why am I so viscerally against helping them?

https://forum.effectivealtruism.org/posts/6bpQTtzfZTmLaJADJ/rebutting-every-objection-to-giving-to-the-shrimp-welfare

This article is an exploration of my explanation to myself. I’m super happy to get feedback and discuss.

My core argument against the general thrust of the article is that shrimp have no moral weight and so their suffering is irrelevant. I think this article is mildly problematic, spreading confused thinking about what is important in the world and diverting resources from more valuable problems to solve.

I also think that this topic is not actually simple and it is very reductionist to make the case “Pain is bad because it hurts”. Sure it's bad for the sufferer, but that's not very helpful in deciding what the “right” course of action is.

It has however prompted quite a bit of very enjoyable research into morality 🙂.

Conclusion

The default position on any topic should be inaction, we do not hold the resource to complete all possible actions and so must prioritize. Moral weight is one way.

Suffering is a negative thing for those affected and should be alleviated if resources are available and the cost of doing so is balanced with the moral weight of the suffering entity.

The Shrimp are suffering, but hold zero moral weight due to total lack of contribution. Inaction is recommended.

Morality

My understanding of morality is that it is

Situational - the level of pro-social action you can perform depends on your available resources and so what is “right” changes.
Grey scale - level of impact on a society changes an entity's moral weight rather than specific criteria leading to inclusion within the boundaries of a moral circle
Not absolute - It is an idea set in the marketplace of ideas like any other, each individual has a version roughly aligned with their society and the societies master set is a kind of average of its members beliefs.
A social technology - its purpose is to convince the individual to be pro - social even when it is negative for the individual. This friction of interests creates a permanent tension within all societies, this can be a source of change.
Individual entities (animals / plants / fictional characters included) have different moral weight depen

...
D&D.Sci Dungeonbuilding: the Dungeon Tournament Evaluation & Ruleset
by aphyer — 2025-01-07T05:02:25.929Z

This is a follow-up to last week month year's D&D.Sci scenario: if you intend to play that, and haven't done so yet, you should do so now before spoiling yourself.

There is a web interactive here you can use to test your answer, and generation code available here if you're interested, or you can read on for the ruleset and scores.

RULESET

Encounters

The following encounters existed:

Encounter Name	Threat (Surprised)	Threat (Alerted)	Alerted By	Tier
Whirling Blade Trap	--	2	--	1
Goblins	1	2	Anything	1
Boulder Trap	--	3	--	2
Orcs	2	4	Anything	2
Clay Golem	--	4	--	3
Hag	3	6	Tier 2 and up	3
Steel Golem	--	5	--	4
Dragon	4	8	Tier 3 and up	4

Each encounter had a Threat that determined how dangerous it was to adventurers. When adventurers encountered that, they would roll [Threat]d2 to determine how challenging they found it.

However, many encounters had two different Threat levels, depending on whether they were alerted to the adventurers or not. (A dragon that's woken up from its slumber, or a hag who's had time to prepare her nastiest spells, or orcs who have gotten their armor on, are much more dangerous than the same encounters when surprised by the adventurers).

Traps and Golems do not rest and cannot be surprised, but Goblins/Orcs/Hag/Dragon encounters could be drastically more/less threatening.

Encounters were alerted by the adventurers having sufficiently loud and dramatic fights in an adjacent room. If a Dragon heard the adventurers fighting a Golem in an adjacent room, it would wake up - if it heard them fighting Goblins in an adjacent room, it would just ignore the Goblin noises, roll over and go back to sleep.

Pathing

Adventurers always took as short a route as possible to the goal:

So they would enter in Room 1, then go to either Room 2/4, then room 3/5/7 (choosing from the two rooms adjacent to their previous choice) and so on.

At each step, an adventuring party scouts the adjacent rooms (using their sneaky Rogue/their invisible Wizard/their powerful Divination magic) and chooses the less threatening one (accounting for alertness, e.g. an alerted Dragon is more threatening than a Steel Golem but an unalerted Dragon is less threatening). If two encounters are equally threatening (e.g both paths have Goblins, or one has alerted Orcs while the other has a sleeping Dragon), they will choose at random.

Score

Depending on how well each fight went, it might do more or less damage: a fight...
Incredibow
by jefftk — 2025-01-07T03:30:02.197Z

Back in 2011 I got sick of breaking the hairs on violin bows and
ordered an

Incredibow. The
hair is polymer filament, and it's very strong. I ordered a 29" Basic
Omnibow, Featherweight, and it's been just what I wanted. I think
I've broken something like three hairs

ever,
despite some rough chopping.

Thirteen years, and it could be new yesterday.

While they don't play identically to traditional bows and it takes a
bit of adjustment, if people had been playing with synthetic hair
historically I really doubt horsehair would take off today.

Comment via: facebook, lesswrong, mastodon, bluesky
Building Big Science from the Bottom-Up: A Fractal Approach to AI Safety
by Lauren Greenspan — 2025-01-07T03:08:51.447Z

Epistemic Status: This post is an attempt to condense some ideas I've been thinking about for quite some time. I took some care grounding the main body of the text, but some parts (particularly the appendix) are pretty off the cuff, and should be treated as such.

The magnitude and scope of the problems related to AI safety have led to an increasingly public discussion about how to address them. Risks of sufficiently advanced AI systems involve unknown unknowns that could impact the global economy, national and personal security, and the way we investigate, innovate, and learn. Clearly, the response from the AI safety community should be as multi-faceted and expansive as the problems it aims to address. In a previous post, we framed fruitful collaborations between applied science, basic science, and governance as trading zones mediated by a safety-relevant boundary object (a safety case sketch) without discussing the scale of the collaborations we were imagining.

As a way of analyzing the local coordination between different scientific cultures, a trading zone can sometimes have a fractal structure; at every level of granularity – a coarse graining of what is meant by ‘local’ – another trading zone appears. In addressing a narrow problem or AI safety sub-goal, like a particular safety case assumption, a small collaboration between different academic groups at an institution (or even across institutions) can constitute a trading zone. Zooming all the way out, the entire field of AI safety can be seen as a trading zone between basic science, applied science, and governance, with a grand mission like ‘create safety cases’ or ‘solve safety’ as its boundary object. This large-scale trading zone may result from (or lead to!) the development of big science for AI safety. Moreover, smaller trading zones would likely benefit from an overarching culture of coordination and resource-sharing that big science would provide.

In this post, I discuss the possible pathways to a big science for AI safety in terms of what is possible, what is likely, and what would be most beneficial, in my view. To ground the discussion, I hold that AI safety should have[1]:

a mosaic scientific culture to maintain the virtues of an a-paradigmatic field.
active engagement between academia, industry, and governance.
a unifying focus that is broad enough to maintain a sufficient r

...
My Experience With A Magnet Implant
by Vale — 2025-01-07T03:01:21.410Z
Link: https://vale.rocks/posts/my-experience-biohacking

TL,DR: I got a magnet implanted in my hand for the purpose of lifting ferrous objects and sensing magnetic fields.

Biohacking is a term that has unfortunately become entangled with tabloid tips, TikTok health trends, and pseudo’science’. However, beneath this surface noise exists dedicated communities exploring the frontiers of human enhancement through do-it-yourself cybernetics, biochemistry, and body modifications. Communities driven by interests in transhumanism and an alignment with the hacker ethos that push boundaries in ways that extend far beyond the conventional ear piercing.

Among the many bleeding-edge modifications that have come about in recent years, magnet implants stood out to me as perhaps the most intriguing, so I went about getting one.

Continues in post
You should delay engineering-heavy research in light of R&D automation
by Daniel Paleka — 2025-01-07T02:11:11.501Z
Link: https://newsletter.danielpaleka.com/p/you-should-delay-engineering-heavy

tl;dr: LLMs rapidly improving at software engineering and math means lots of projects are better off as Google Docs until your AI agent intern can implement them.

Implementation keeps getting cheaper

Writing research code has gotten a lot faster over the past few years. Since 2021 and OpenAI Codex, new models and tools such as Cursor built around them have saved myself more and more time on coding every year.

This trend is accelerating fast: AI agents using Claude-3.5-Sonnet and o1-preview can do tasks that take ML researchers up to 2 hours of coding. This is without considering newer models such as o3, which do 70% on SWE-bench out of the box.
Yet this progress remains somewhat concentrated in implementation: progress on “soft” skills like idea generation has, as far as I can tell, been slower.

I’ve come to believe that, if you work in technical AI safety research, this trend is a very practical consideration that should be the highest order bit in your decisions on what to spend time on.

Hence, my New Year's resolution is the following: Do not work on a bigger project if there is not a clear reason for doing it now. Disregarding AGI timelines [1] 1, the R&D acceleration is a clear argument against technical work where the impact does not critically depend on timing.

When later means better

The wait calculation in space travel is a cool intuition pump for today’s AI research. In short, when technological progress is sufficiently rapid, later projects can overtake earlier ones.

For instance, a space probe sent to Alpha Centauri in 2025 will likely reach there after the one sent in 2040, due to advances in propulsion technology. Similarly, starting a multi-year LLM training run in 2022 would not have yielded a better model than starting a much shorter training run in 2024. [2]

The above examples involve long feedback loops, and it’s clear why locking in too early has issues: path dependence is high, and the tech improves quickly.

Now, my research (and likely your research too) has had much faster feedback loops, and path dependence in research projects is not that high if LLMs can refactor the codebase. However, it still does not make marginal sense to start some projects now if those can be done later.

If you work in AI safety, a common issue is having a lot of half-baked project ideas that you'd like to do and too...