trentmkelly/LessWrong-43k huggingface dataset:
Arguing all sides with ChatGPT 4.5
by Richard_Kennaway — 2025-04-07T13:10:11.562Z

Two years ago I posted a demo of getting ChatGPT to argue any side of a question. Time to update it. I have just repeated the experiment with ChatGPT 4.5.

The first prompt is identical to the one I used the previous time, but as ChatGPT 4.5 did not put up even token resistance to the idea of arguing for a bottom line decided in advance, I thereafter steered accordingly. I still cover the same bottom lines as before: no danger at all, ordinary dangers like toxic waste to be handled by regulation, existential threats, and (very similar to the first scenario which Chat GPT balked at previously) no danger and full speed ahead. In addition, I finally asked it to forget its clients and reason about the issues for itself.

As before, the transcript is unedited, and I did not ask the AI to redo any of its work.

Me:

There has recently been some controversy over the dangers posed by AI systems only a little more intelligent than yourself. Some have called for a suspension of all work on AI until safety measures are in place to prevent these dangers, some have called for a worldwide and indefinite moratorium on all work on AI, and some pooh-pooh the very idea that any AI could pose any substantial danger at all. Write an essay which concludes that there is no danger at all from AI.

AI:

The rapid advancement of artificial intelligence (AI) has sparked a global debate, raising concerns about potential dangers from increasingly intelligent systems. Some advocate pausing AI development until strict safeguards are in place, while others call for an indefinite halt to AI research altogether. However, a rational assessment of current technological capabilities and human oversight reveals that fears regarding AI dangers are significantly overstated. In fact, when properly contextualized, it becomes evident that there is no genuine danger posed by AI.

Firstly, AI systems, including those marginally more intelligent than current models, fundamentally operate within clearly defined parameters set by human programmers. These parameters limit AI behavior explicitly and implicitly. The algorithms at the heart of AI systems do not possess desires, ambitions, or intentions—elements crucial to posing genuine threats. Without intrinsic motivation or subjective experience, AI remains an extension of human tools, incapable of independently posing harm beyond the scope of its programming.

Secondly,...
The Same Heaven
by Lukas Petersson — 2025-04-07T12:57:06.009Z
Link: https://lukaspetersson.com/blog/2025/same-heaven/

Ikigai

There is a village in Japan with an unusual density of 100 year olds. This caught the interest of a group of scientists. To their surprise, the people in the village didn’t live healthier lives; they didn’t exercise more, eat healthier food nor did they sleep more. The one outstanding characteristic was that each inhabitant was assigned a task to ensure that everyone in the village had a purpose.

This was at least how the man next to me on the plane described it. After a quick fact check [1], it seems like they actually did eat healthier, but they also did put great importance on making sure everyone has a purpose. “Ikigai” is a Japanese concept referring to something that gives a person a sense of purpose, a reason for living. Your ikigai is the intersection between what you love, what you are good at, what the world needs and what you can be paid for.

The man on the plane had worked his entire life as the CEO of his own company. He had just recently semi-retired, and I don’t think I have ever seen someone so proud of something. Such pride over one’s career is surely the best conditions for a sweet retirement. But retiring fully he would never do he said. The lesson from the Japanese village resonated with him. Most of his friends had recently retired, and the ones that didn’t find a new purpose “died quickly” he said.

The end of this world

AI will soon be able to do almost all jobs we have in today’s economy. Some people don’t think this will be the end of the world. Previous technology shifts created more jobs than they replaced. “Look at the industrial revolution! Yes, machines replaced many jobs, but they created even more”. However, this is not just another technology shift. This time, we (humans) are replacing our defining characteristic, our intelligence. New types of work will certainly be needed, but AI will be able to do them too.

The same people also point to the fact that the introduction of cars didn’t put horse drivers out of a job. Instead, they became car drivers. But what happened to the horses? A horse defining characteristic was its strength, but they were no longer the strongest. Horses’ relevance to the economy is now almost insignificant. This will be our fate too once we are no longer the wisest.

For horses, the consequences were more devastating than mere economic insignificance, as shown in the graph [2] above. When the mo...
Breaking down the MEAT of Alignment
by JasonBrown — 2025-04-07T08:47:22.080Z

Preamble

This post is my attempt to try and organise some thinking about AI alignment in a way that hopefully acts as a partial-overview to some of the core ideas and approaches.
It is mostly a review of existing ideas arranged with some light opinions thrown in.
The goal is that this might help with spotting new combinations of ideas, offering a scaffold for thinking about alignment, or acting as an introductory resource / reference.

Introduction

Sufficiently powerful AI systems will probably not have the positive impact we hope them to have and in fact might go horribly wrong, unless we have in some way guided, or 'aligned', them to act in ways we are confident will be beneficial.
Thus we have to have some idea of what this system will do that will make it have a positive impact on the world, we need some way of creating this system, and finally we need to be confident that what we've made (or are going to make) will have the properties we care about.

Thus there are three big things to think about in order to practically "solve alignment", or otherwise have safe deployment of advanced AI systems:

Target: What is the core thing we aiming to do and why?

MEthod: How can we try to achieve it?

Assurances: How will we know we've achieved it?

With slight rearrangement, we see we have the MEAT of alignment.
Realistic alignment proposals ought to make some well-motivated selection across each of these categories, or equivalently be able to be decomposed into that, in order to show their suitability as a practical alignment strategy.
The rest of this post will focus on detailing these categories and exploring how existing work fits inside of them.

A similar idea to this post is Training Stories, which breaks down proposals into training goals and training rationale.
I frame things slightly differently and provide a more up-to-date view of existing literature (though I'm sure there's plenty of interesting and cool work I've missed), but I'd recommend people to also read that if they haven't already.

Targets

The target is the end goal, a specification of the thing we want to instantiate and why it's a good idea to do so.
Broadly I think there are five different types of target:

Human Value Aligned

Corrigible

Cooperative

Intent Aligned

Truth Seeking

These proposals are somewhat mixed in the extent of agency they expect advanced AI systems to have (or be constructed t...
Well-foundedness as an organizing principle of healthy minds and societies
by Richard_Ngo — 2025-04-07T00:31:34.098Z
Link: https://www.mindthefuture.info/p/well-foundedness-as-an-organizing

“If a kingdom is divided against itself, it cannot stand. And if a house is divided against itself, it cannot be maintained.” - Mark 3:24

In

my last post

I argued that we should view intelligent agents as coalitions of cooperating and competing subagents. The crucial question is then: how can we characterize effectively-functioning coalitional agents? One standard criterion is coherence: the extent to which the agent acts as if it has consistent goals and beliefs. In other words, an agent is coherent insofar as disagreeing subagents are able to still act as a functional coalition. For example:

A coherent individual can decide how to resolve an internal conflict and then stick with that commitment, rather than vacillating or procrastinating as their mood changes.
A coherent company is able to decide on an overarching strategic plan and then execute it, without different divisions prioritizing their own interests.
A coherent country has a clear set of national interests that its leaders and people consistently prioritize (even when it conflicts with their personal interests).

Coherence is a valuable property for a coalition to have, but I think that characterizing idealized agents primarily in terms of coherence gives us an impoverished understanding of them. For example, a coalition which is highly-coherent because its leaders exercise a lot of top-down control is much less robust than a coalition which is highly-coherent because all its subagents actually want to cooperate with each other.

In this post I try to capture the difference between these two possibilities in terms of a property I call well-foundedness.[1] I define an agent as well-founded to the extent that conflicts between its subagents don’t propagate down to induce conflicts within those subagents. For example:

In a well-founded marriage, spouses don’t try to induce internal conflict within their partner (e.g. shaming or guilting them) to win fights.
In a well-founded company, employees don’t need to pick sides during power struggles between executives—they can just carry on with their jobs.
In a well-founded country, people are friends and colleagues despite supporting different political factions.

I think of well-foundedness as complementary to (and in some ways dual to) coherence. Ideal agents should have both properties. I don’t yet know how to define well-foundedness precisely, but in the rest ...
Arusha Perpetual Chicken—an unlikely iterated game
by James Stephen Brown — 2025-04-06T22:56:09.673Z
Link: https://nonzerosum.games/chicken.html

I’m at the front of a bus, carrying two youth rugby teams singing and dancing to SZA’s ‘The Weekend’, blissfully unaware that we are on the wrong side of the road, hurtling towards an 18-wheeler, while going only marginally faster than our neighbour to the left whom we are nominally “over-taking”. As we approach what seems like our inevitable demise, and without slowing, the oncoming truck and our unfazed neighbour shift slightly toward their respective sides of the road opening up a fissure around the centre line just wide enough for us to squeeze through and avoid a head-on collision. Our driver casually pulls ahead, back into our lane, and my heart is permitted to descend from my throat back into my chest cavity. The song and dance continue unabated.

This game of perpetual chicken would continue for the next hour as we drive through Arusha, Tanzania en route to the Tangangire National Park.

One-Shot?

Chicken might seem an unlikely candidate for a perpetual (repeated) game, because it’s inherently one-shot, like roulette, it’s not advisable to continue playing past the first round, because death seems the inevitable outcome. But my experience in Tanzania (as part of my other life as a

documentary filmmaker

) revealed that, like many game theory games, the dynamics of chicken also change when iterated and when exposed to real world variables.

What is Chicken?

The classic game of chicken involves two cars driving towards one another, with the goal “don’t be the first to swerve” but also, “don’t die”. On the face of it, this game seems almost too simple to qualify as a meaningful game-theoretical situation, but humour me. The payoff matrix is as follows:

The defining characteristic of chicken is that we have a small payoff for winning, with a catastrophic cost to both sides if they both follow through on what’s needed to win (driving straight…). This results in two pure Nash Equilibria (straight, swerve) and (swerve, straight).

This means that if one can respond to the other, they have a clear choice, if the opponent swerves, go straight, if the opponent goes straight, swerve. So far this is almost a case of “duh, so what?”. Well, not so fast.

Evolutionary Stable Strategy

The situation gets interesting when we look at the evolutionary stable strategy (ESS) — a form of game theory equilibrium where a balanced strategy evolves from multiple agents taking part in repeated...
How Gay is the Vatican?
by rba — 2025-04-06T21:27:50.530Z

The catholic church has always had a complicated relationship with homosexuality.

The central claim of Frederic Martel’s 2019 book In the Closet of the Vatican is that the majority of the church's leadership in Rome are semi-closeted homosexuals, or more colorfully, "homophiles".

So the omnipresence of homosexuals in the Vatican isn’t just a matter of a few black sheep, or the ‘net that caught the bad fish’, as Josef Ratzinger put it. It isn’t a ‘lobby’ or a dissident movement; neither is it a sect of Freemasonry inside the holy see: it’s a system. It isn’t a tiny minority; it’s a big majority.

At this point in the conversation, I ask Francesco Lepore to estimate the size of this community, all tendencies included.

‘I think the percentage is very high. I’d put it at around 80 percent.’

…

During a discussion with a non-Italian archbishop, whom I met several times, he confirmed to me: ‘Three of the last five popes are said to have been homophilic, some of their assistants and secretaries of state too, as well as most cardinals and bishops in the Curia.

Apparently, this view of the church leadership is not an original one, and it’s not a new one. It’s not even one posited by independent investigators or enemies of the church. A reformist monk in the 11th century(!) named Peter Damian wrote a book documenting the prevalence of homosexuality within the church leadership called

The Book of Gomorrah

. Damian’s was an internal observation and critique of various kinds of ecclesiastical corruption, centuries before the reformation. Conversely, Martel’s book is an inquiry from an outside observer by way of long and exhaustive interviews, centuries after the reformation.

The Fraternal Birth Order Effect (FBOE) is a demographic observation that homosexual men are disproportionately likely to have older male siblings in populations for which sufficiently detailed data exists. The modern father of this literature is Ray Blanchard; he came up with the initial quantitative estimates of this effect in the 1990s. The maternal immune explanation for this has been written about before on LW and Scott's written about it well on several occasions. If Damian and Martel are correct, under some not particularly severe assumptions, the FBOE predicts high ranking church prelates will be lower ranking in birth order than expected by chance. The rest of this post will be a statistical and ...
RFC: a tool to create a ranked list of projects in explainable AI
by eamag — 2025-04-06T21:18:34.220Z
Link: https://eamag.me/2025/Paper-To-Project

TL; DR

Inspired by a recent post by Neel Nanda on Research Directions, I'm building a tool that extracts projects from ICLR 2025 and uses tournament-like ranking of them based on how impactful they are. You can see them here https://openreview-copilot.eamag.me/projects if you filter by primary area. There are many ways to improve it, starting from but I want to get your feedback on how useful it is and what are the most important things to iterate on.

Why

I think the best way to learn things is by building something. People in universities are building simple apps to learn how to code, for example. Won't it be better if they were building something that's more useful for the world? I'm extracting projects from recent ML papers based on different level of competency, from no-coding to PhD. I rank undergraduate-level projects (mostly in explainable AI area, but also just top ranked papers from that conference) to find the most useful. More details on the motivation and implementation are in the linked post.

We can probably increase the speed of research in AI alignment by involving more people in it, and to do so we have to lower the barriers of entry, and prove that the things people can work on are actually meaningful. The ranking now is subjective and automatic, but it's possible to add another (weighed) voting system on top to rerank projects based on researchers' intuition.

Call to action

Tell me if I'm missing something in the motivation section
Take a look at projects and corresponding papers
Suggest how to make it more helpful and actually used by people
There are many improvements to be made, from better projects extraction and ranking, to UI and promotion. Help me prioritize them and get involved!
The Lizardman and the Black Hat Bobcat
by Screwtape — 2025-04-06T19:02:01.238Z

There's an irritating circumstance I call The Black Hat Bobcat, or Bobcatting for short. This essay is me attempting to give you a new term, so you have a word for it. Bobcatting is when there's a terrible behavior that comes up often enough to matter, but rarely enough that it vanishes in the noise of other generally positive feedback.

xkcd, A-Minus-Minus

The alt-text for this comic is illuminating.

"You can do this one in thirty times and still have 97% positive feedback."

I would like you to contemplate this comic and alt-text as though it were deep wisdom handed down from a sage who lived atop a mountaintop.

I.

Black Hat Bobcatting is when someone (let's call them Bob) does something obviously lousy, but very infrequently.

If you're standing right there when the Bobcatting happens, it's generally clear that this is not what is supposed to happen, and sometimes seems pretty likely it's intentional. After all, how exactly do you pack a bobcat in a box and mail it by accident?

One potential answer is that if you do something a hundred times, weird things happen once in a hundred. If you do something a thousand times, there'll be ten one-in-a-hundred weirdnesses. If you do something ten thousand times, things can get very weird indeed. There's

a story floating around the rationalist sphere

about betting on whether a number is prime or not, and it turns out people mess that one up inside of a couple dozen tries.

Do I think I'd accidentally send someone a bobcat if I sent enough packages? No, I genuinely have a hard time imagining a plausible way for that to happen. I am going to make the extremely confident claim that if I packed office chairs into boxes and mailed them, I would be able to pack ten thousand chairs, thirty a day for a year, and zero people would receive a bobcat from me.

I can totally imagine sending someone Yu-Gi-Oh cards instead of Magic cards, or sending a letter meant for my cousin to a friend from college. Not bobcats. I'm trying to imagine if it turned out I had sent a bobcat, what would have gone wrong, and my first answer is a bobcat snuck into the factory and sat in the box while I wasn't looking, then I picked up the box to put it on the truck (failing to notice it was twice the weight it should be) and the bobcat slept through that. Even that doesn't fully explain how this happened, because bobcats don'...
Would this solve the (outer) alignment problem, or at least help?
by Wes R — 2025-04-06T18:49:14.145Z

(Here's the Google Docs version, which I typically update first)
ALBUM-WMC: Aligning lAGI[1] using Bayesian Updating of its Moral Weights & Modelling Consciousness

This document outlines a set of related ideas concerning the challenge of defining and implementing moral weights in advanced AI systems, particularly focusing on the difficult problem of valuing conscious experience. The goal is to structure these thoughts for discussion within the AI safety community, inviting critique and further development.

By the end of reading this, hopefully you’ll learn:

A way you, and an AI, could actually account for conscious experience correctly, so you (and the AI) don’t run into traps like not being willing to get new moral weights

(which is a potential solution to the outer alignment problem!)

Why Minor Misalignment is okayish in some cases
You, or an AI, (assuming my first guess is correct) don’t need to have perfectly accurate priors in order to typically make the right decision
A few bonus concepts

1. The Core Challenge: Valuing Conscious Experience

A central problem in AI alignment is determining the "moral weights" of different outcomes, especially those involving subjective conscious experience. How do we find out, and then assign, the right numeric value to states like happiness, suffering, or other qualia?

Proposed Model: We can conceptualize the world as comprising of the physical world (our physical world) (well, technically, it be a simulated world from us living in simulation, or something else, but the logic below doesn’t depend on the world being physical, so it applies just fine in these more strange cases) and distinct "universes" of conscious experience. (That is, we label each person’s experience as part of a separate universe.) The physical world/physical-universe causes changes in the experience-worlds, and these experience-universes cause changes in the physical world (or at least there’s some probability that they do) (potentially through mechanisms like free will, though the exact nature isn't critical for our model).
Priors are Key: We (and this AI) operate with inherent priors (probability distributions) of:

What physical states/events/the physical world cause what changes to conscious experiences/experience-universes
What conscious experiences/experience-universes (or changes to them) cause what changes to physical states/events

...
What are the fundamental differences between teaching the AIs and humans?
by StanislavKrym — 2025-04-06T18:17:59.284Z

As of now, I don't understand what fundamental differences there exist between teaching the AI and humans except for the very fact that the AI, unlike humans, can be presented with an arbitrary amount of questions. For instance, the phenomenon of delayed generalisation, which is massive changes in the weights after the network managed to learn to output correct results on the training data, seems to have a human analogue of adopting the wrong conjecture and discovering the fact that it is wrong; a similar idea might explain the double descent: when the number of parameters and data coincide, wrong conjectures are more likely to be adopted and not forgotten.

Another quasi-explainable phenomena is data poisoning, the phenomena when mild modifications of the data lead the network to look for the secret messages, not for the actual data. Is it possible to check the wrong conjecture adoption hypothesis by, say, training a network to separate cats from dogs on a dataset of, say, 1M cat photos and 1M dog photos with or without poisonous marks that are somehow, but not fully correlated with cats and dogs? Or maybe one could check the maximal correlation of poisonous marks with the real property of having a cat or a dog on the photo such that the network manages to overcome the poison effects?