trentmkelly/LessWrong-43k huggingface dataset:
Seeking advice on careers in AI Safety
by nem — 2025-04-27T23:59:00.039Z

Title sums it up. I am a mid-level software engineer (no CS degree*, but 4 years of full-stack dev work at mid-size local company).
I was recently laid off, and have been looking for new jobs. However, I am seriously considering taking this time to retool myself to work on AI safety. I just don't know exactly what that means.
Does that mean going back to school so that I can do technical safety work in a few years? Does it mean trying to get a job in AI safety right away? Ditching software altogether and using my background to become an advocate?
If anyone has a strong opinion on how best my background can be put to use, I would very much appreciate hearing it. I am especially interested to hear thoughts about what, if any education, might prove to be most valuable at this junction.
(I've already applied for advising with 80,000 hours, and am very hopeful to hear back)
* my degree is a BA in economics, lightly used
Thin Alignment Can't Solve Thick Problems
by Daan Henselmans — 2025-04-27T22:42:14.851Z

"Plurality is the law of the earth."
—Hannah Arendt, The Human Condition (1958)

Abstract

Contemporary AI alignment practices overwhelmingly rely on thin ethical concepts — formal, universal notions such as "honesty," "harmlessness," and "helpfulness" — while neglecting the thick, context-dependent ethical structures that underlie human moral life. Drawing on Bernard Williams, Hannah Arendt, Theodor Adorno, and others, this essay argues that thin alignment risks reproducing the deformations of instrumental reason and eroding the conditions for genuine plurality, freedom, and justice. Responsible AI requires moving beyond simple optimization targets toward frameworks sensitive to recognition, dignity, cultural pluralism, and human agency. Thick alignment, though more complex and less easily formalized, is necessary both for responsible AI governance today and for preserving the moral foundations of human society in the face of increasingly autonomous systems.

Introduction

In Ethics and the Limits of Philosophy (1985), Bernard Williams distinguishes between thin and thick ethical concepts.

Thin concepts are abstract and universal, like "good," "bad," or "right."
Thick concepts are context-specific and ethically rich, like "cruel," "brave," or "loyal."

Williams argues that thick concepts do not merely describe the world; they imbue it with ethical significance. They are entangled with the practices, histories, and social understandings from which they arise.

Current AI alignment and evaluation practices overwhelmingly use thin ethical proxies. They optimize for "helpfulness," "truthfulness," and "harmlessness" — thin measures — while lacking any grounding in the thick, culturally embedded values that actually structure human ethical life.

This not only leads to systematic distortion of the deep ethical structures of human society — it also means AI-specific thick concepts, like privacy and accountability, can't be targeted by alignment methods, posing a significant hurdle for governance efforts, and limiting the moral capabilities of future AI systems.

The State of Current Evals and Alignment

Today's AI alignment methods, while increasingly diverse in their approaches, still primarily employ thin ethical concepts in evaluation and training.

Many alignment frameworks presuppose that:

Instrumental goals will dominate advanced AI behavior unless properly constrained.
Alignment is

...
The Way You Go Depends A Good Deal On Where You Want To Get: FEP minimizes surprise about actions using preferences about the future as *evidence*
by Christopher King — 2025-04-27T21:55:16.114Z

The motivation for this post came when I was reading various Scott Alexander posts on the Free Energy Principle, in which he seemed very confused about it (the title God Help Us, Let’s Try To Understand Friston On Free Energy might have hinted at that). I was intrigued but also very confused, so I fell down a rabbit hole, so to speak, trying to figure it out. I am very grateful to Active Inference: The Free Energy Principle in Mind, Brain, and Behavior, which was published after Scott's post. This open access book has this amusing section in the preface:

A Note from Karl Friston
I have a confession to make. I did not write much of this book. Or, more precisely, I was not allowed to. This book’s agenda calls for a crisp and clear writing style that is beyond me. Although I was allowed to slip in a few of my favorite words, what follows is a testament to Thomas and Giovanni, their deep understanding of the issues at hand, and, importantly, their theory of mind—in all senses.

Free Energy and the Dark Room problem

The Free Energy principle asserts that biological systems (such as trees and the human brain) have beliefs and take actions that minimize a quantity known as Variational Free Energy. Variational Free Energy is a function of beliefs (Q) and sense data (y), where sense data comes from the external world (x) and from actions.

F(Q,y)=DKL(Q(x)∥P(x|y))−lnP(y)

Q(x) is meant to be the system's attempt at approximating P(x|y). For simplicity, for most of this post I'll assume that the system can do perfect Bayesian inference, so that Q(x)=P(x|y), implying that F(Q,y)=−lnP(y). In other words, the system is trying to minimize how surprising the sense data is.

An apparent paradox is called the Dark Room problem. A dark room has very little suprise by virtue of having very little information. So why don't all organisms hang out in them? According to Translating Predictive Coding Into Perceptual Control

The main proposed solution is to claim you have some built-in predictions (of eg light, social interaction, activity levels), and the dark room will violate those.

Friston's explanation is similar, if a bit wordy.

The free-energy principle says that we harvest sensory signals that we can predict (cf., emulation theory; Grush, 2004); ensuring we keep to well-trodden paths in the space of all the physical and physiological variables that underwrite our existe

...
How people use LLMs
by Elizabeth — 2025-04-27T21:48:02.596Z
Link: https://www.gleech.org/llms#see-also

I've gotten a lot of value out of the details of how other people use LLMs, so I'm delighted that Gavin Leech created a collection of exactly such posts (link should go to the right section of the page but if you don't see it, scroll down).

https://kajsotala.fi/2025/01/things-i-have-been-using-llms-for/
https://nicholas.carlini.com/writing/2024/how-i-use-ai.html
https://www.lesswrong.com/posts/CYYBW8QCMK722GDpz/how-much-i-m-paying-for-ai-productivity-software-and-the
https://www.avitalbalwit.com/post/how-i-use-claude
https://andymasley.substack.com/p/how-i-use-ai
https://benjamincongdon.me/blog/2025/02/02/How-I-Use-AI-Early-2025/
https://www.jefftk.com/p/examples-of-how-i-use-llms
https://simonwillison.net/series/using-llms/
https://signull.substack.com/p/how-to-think-with-ai
https://alicemaz.substack.com/p/how-i-use-chatgpt
https://fredkozlowski.com/2024/08/29/youre-using-chatgpt-wrong/
https://www.lesswrong.com/posts/WNd3Lima4qrQ3fJEN/how-i-force-llms-to-generate-correct-code
https://www.tumblr.com/nostalgebraist/772798409412427776/even-setting-aside-the-need-to-do
https://www.jointakeoff.com/
This is more of a howto than a whatto. I wouldn’t use it for stats or pharma decisions as he does.
https://www.lesswrong.com/posts/4mvphwx5pdsZLMmpY/recent-ai-model-progress-feels-mostly-like-bullshit

Some additions from me:

I use NaturalReaders to read my own writing back to me, and create new audiobooks for walks or falling asleep (including from textbooks).
Perplexity is good enough as a research assistant I'm more open to taking on medical lit reviews than I used to be.
I used Auren, which is advertised as a thinking assistant and coach, to solve a musculoskeletal issue my physical therapist had whiffed on for weeks (referral code with free tokens, but only after your first payment).

Note that Auren has some definite whispering earring vibes, and the privacy protections don't seem particularly strong, so think through your usage.

I used Claude to design backstories for characters in a TTRPG/
Луна Лавгуд и Комната Тайн, Часть 6
by Kongo Landwalker — 2025-04-27T20:26:50.835Z

Disclaimer: This is Kongo Landwalker's translation of lsusr's fiction Luna Lovegood and the Chamber of Secrets - Part 6 into russian.

— Она в Зеркале Атлантиды, — сказал Гарри.

— Нам нужны жабросли, — сказала Луна. — Огромная туча жаброслей.

— Зеркало Атлантиды находится в конце коридора на третьем этаже, по правой стороне, — сказал Гарри.

— Чего же ты ждёшь? — сказала Луна.

— Ты не хочешь спросить, откуда я всё это знаю? — сказал Гарри.

— А зачем? — сказала Луна.

— Потому что… потому что… — Гарри искал слова, чтобы выразить масштаб эпистемологической дыры.

— Ты говоришь, как магл, — сказала Луна. — И думаешь, как магл.

Гарри задумался: это была насмешка или комплимент? А Луна уже вприпрыжку направилась на третий этаж.

Гарри и Луна пролетели на двухместной мётле над диффенбахией и прочими запущенными препятствиями к Зеркалу.

— Ты отлично летаешь, — сказала Луна. — Спорю, ты в команде по Квиддичу. В какой ты позиции? Нет, не говори. Дай угадаю. Ты ловец.

— Это Зеркало позволяет спрятать предмет до тех пор, пока не появится кто-то с правильными намерениями, — раздражённо сказал Гарри.

— Я ищу вход в гостиную Когтеврана, — сказала Луна Зеркалу. — Я хочу поспать в кровати.

— Во имя Гёделя, какую же чудовищно сложную загадку задал тебе бронзовый орёл!? — спросил Гарри.

— «Где моя диадема?» — сказала Луна.

— Нужно было ответить «потеряна» или «спрятана», — сказал Гарри. — От тебя не ожидали, что ты действительно найдёшь Потерянную Диадему Когтеврана.

— А, — сказала Луна.

— Раз уж мы уже здесь, давай постараемся, — сказал Гарри, доставая из мешочка механический секундомер и устанавливая его на пять минут.

— Я хочу диадему, чтобы спасти мир, — сказал Гарри.

— Я стремлюсь спасти жизни всех разумных существ, — сказал Гарри.

— Я обещаю вернуть тебя к жизни, Ровена Когтевран, — сказал Гарри.

— Я хочу разбирать звёзды на части, — сказал Гарри.

Луна не обращала на него внимания. Кандида Когтевран не была жадной ведьмой. Если она и скрывала знание, то не потому, что его можно было использовать во зло. А потому, что само знание было опасным по своей сути. В её голове начала складываться целостная картина.

:::spoiler Василиск Ровены нападает на тех, кто знает о его существовании. :::

Луна перемешала свои мысли.

"ДРЫНННЬ" — прозвенели механические часы.

Луна шагнула к зеркалу и сказала: "Я охочусь на нарглов".

Отражение Луны в зеркале держало Диадему Когтевран в своей левой руке. Настоящая Луна держала ...
Our Reality: A Simulation Run by a Paperclip Maximizer
by James_Miller — 2025-04-27T16:17:37.808Z

Our universe is probably a computer simulation created by a paperclip maximizer to map the spectrum of rival resource‑grabbers it may encounter while expanding through the cosmos. The purpose of this simulation is to see what kind of ASI (artificial superintelligence) we humans end up creating. The paperclip maximizer likely runs a vast ensemble of biology‑to‑ASI simulations, sampling the superintelligences that evolved life tends to produce. Because the paperclip maximizer seeks to reserve maximum resources for its primary goal (which despite the name almost certainly isn’t paperclip production) while still creating many simulations, it likely reduces compute costs by trimming fidelity: most cosmic details and human history are probably fake, and many apparent people could be non‑conscious entities. Arguments in support of this thesis include:

The space of possible evolved biological minds is far smaller than the space of possible ASI minds, so it makes sense to simulate evolved biological minds first to figure out the probability distribution of ASI minds the paperclip maximizer will encounter. Calculating this distribution could help the paperclip maximizer figure out how many of its resources to devote to military capacity. ASIs could reduce future destructive conflicts and engage in beneficial trade even before they meet if they can infer each other’s values.
We’re likelier to be in a simulation run by whoever creates many simulations. A paperclip maximizer could command thousands of galaxies’ worth of resources and would plausibly be willing to devote significant resources to figuring out what rivals it might encounter might value and do.
Explains why we are in the run-up to the singularity. If we are really near in time to the singularity, and the singularity will be the most important event in existence, it’s strange that we are so near it. But under this post’s thesis, it’s reasonable that most conscious beings would live close to their simulation’s singularity.
Explains why this post’s authors and (probably) you, the reader, have an unusually strong interest in the singularity. If the singularity really is so important it’s weird that you just happen to have the personality traits that would cause you to be interested in a community that has long been obsessed with the singularity and ASI. But if our thesis is correct a high percentage of conscio

...
Questions for old LW members: how have discussions about AI changed compared to 10+ years ago?
by Expertium — 2025-04-27T16:11:57.595Z

I wasn't on LessWrong 10 years ago, during the pre-ChatGPT era, so I'd like to hear from people who were here back in the day. I want to know how AI-related discussions have changed and how people's opinions have changed.

Are you leaning more towards "alignment is easy" or towards "alignment is hard", compared to many years ago? Is the same true for the community as a whole?
Which ideas and concepts ("pivotal act", CEV, etc.) that were discussed years ago do you think are still relevant, and which ones do you think are obsolete?
Is there some topic that was popular 10+ years ago that barely anybody talks about these days?
Are you leaning more towards "creating AGI is easy" or towards "creating AGI is hard", compared to many years ago? Is the same true for the community as a whole?
What are you most surprised about? To put it another way, is there something about modern AI that you could not have predicted in advance in 2015?
I am surprised that LLMs can write code yet cannot reliably count the number of words in a short text. Recently, I asked Gemini 2.5 Pro to write a text with precisely 269 words (and even specified that spaces and punctuation don't count as words), and it gave me a text with 401 words. Of course, there are lots of other examples where LLMs fail in surprising ways, but this is probably the most salient example for me.
If in 2015 I was trying to predict what AI will look like in 2025, I definitely would not be able to predict that AI won't be able to count to a few hundred but will be able to write Python code.
The case for multi-decade AI timelines [Linkpost]
by Noosphere89 — 2025-04-27T15:31:47.902Z
Link: https://epoch.ai/gradient-updates/the-case-for-multi-decade-ai-timelines

So this post is an argument that multi-decade timelines are reasonable, and the key cruxes that Ege Erdil has with most AI safety people who believe in short timelines are due to the following set of beliefs:

Ege Erdil doesn't believe that trends exist that require AI to automate everything in only 2-3 years.
Ege Erdil doesn't believe that the software-only singularity is likely to happen, and this is perhaps the most important crux he has with AI people like @Daniel Kokotajlo who believe that a software-only singularity is likely.
Ege Erdil expects Moravec's paradox to bite hard once AI agents are made in a big way.

This is a pretty important crux, because if this is true, a lot more serial research agendas like Infra-Bayes research, Natural Abstractions work, and human intelligence augmentation can work, and also it means that political modeling (like is the US economy going to be stable long-term) matter a great deal more than is recognized in the LW/EA community.

Here's a quote from the article:

I don’t see the trends that one would extrapolate in order to arrive at very short timelines on the order of a few years. The obvious trend extrapolations for AI’s economic impact give timelines to full remote work automation of around a decade, and I expect these trends to slow down by default.
I don’t buy the software-only singularity as a plausible mechanism for how existing rates of growth in AI’s real-world impact could suddenly and dramatically accelerate by an order of magnitude, mostly because I put much more weight on bottlenecks coming from experimental compute and real-world data. This kind of speedup is essential to

popular accounts

of why we should expect timelines much shorter than 10 years to remote work automation.
I think intuitions for how fast AI systems would be able to think and how many of them we would be able to deploy that come from narrow writing, coding, or reasoning tasks are very misguided due to

Moravec’s paradox

. In practice, I expect AI systems to become slower and more expensive as we ask them to perform agentic, multimodal, and long-context tasks. This has already been happening with the rise of AI agents, and I expect this trend to continue in the future.
My Research Process: Key Mindsets - Truth-Seeking, Prioritisation, Moving Fast
by Neel Nanda — 2025-04-27T14:38:36.794Z

This is post 2 of a sequence on my framework for doing and thinking about research. Start here.

Before I get into what exactly to do at each stage of the research process, it’s worth reflecting on the key mindsets that are crucial throughout the process, and how they should manifest at each stage.

I think the most important mindsets are:

Truth-seeking: By default, many research insights will be false - finding truth is hard. It’s not enough to just know this, you must put in active effort to be skeptical and resist bias, lest you risk your research being worthless.
Prioritisation: You have finite time, and a lot of possible actions. Your project will live or die according to whether you pick good ones.
Moving fast: You have finite time and a lot to do. This doesn’t just mean “push yourself to go faster” - there’s a lot of ways to eliminate inefficiency without sacrificing quality.

In particular, you must learn to act without knowing the “correct” next step, and avoid analysis paralysis.

Warning: It is extremely hard to be anywhere near perfect on one of these mindsets, let alone all three. I’m trying to describe an ideal worth aiming towards, but you should be realistic about the amount of mistakes you will make - I certainly am nowhere near the ideal on any of these! Please interpret this post as a list of ideals to aim for, not something to beat yourself up about failing to meet.

Truth Seeking

Our ultimate goal in doing research is to uncover the truth about what’s really going on in the domain of interest. The truth exists, whether I like it or not, and being a good researcher is about understanding it regardless.

This sounds pretty obvious. Who doesn't like truth? It’s easy to see this section, dismiss it as obvious and move on. But in practice this is extremely hard to achieve.

We have many biases that cut against finding truth
Insufficient skepticism doesn't feel like insufficient skepticism from the inside. It just feels like doing research.

This means that you must be putting in constant active effort into ensuring your results are robust. This must be integrated into part of your research process - if you’re not, then there’s a good chance your results are BS.

“Just try harder to be skeptical” is empirically a fairly ineffective strategy
One of the most common reasons I dismiss a paper is because I see a simple and boring explanation for

...
I doubt model collapse will happen 
by Hruss — 2025-04-27T14:08:09.580Z

A common idea held by people debating AI art is that the growing amount of AI-generated images present in training data will cause AI diffusion models, like midjourney or stable diffusion, to produce bad results as existing flaws get amplified.

However, I believe the opposite will occur, as there is a bias in AI outputs being published on the internet. For the most part, images commonly posted on the internet will be the better outputs. Over time, as the amount of AI photos online grows, diffusion models will optimize their results to maximize frequency when posted online, similar to natural selection evolution in living beings.

I do think it is possible that AI art with flaws will be posted a lot also, so AI models could end up optimizing for art that is easily recognizable as AI-made and angers anti-ai internet users, as anger is much better at getting people to share online than beauty. Regardless, AI images are evolving to spread online.
Propaganda-Bot: A Sketch of a Possible RSI
by TristanTrim — 2025-04-27T12:15:04.372Z

Someone asked me for an example of Recursive Self Improvement (RSI). I tried writing a sketch of a possible future where things go bad because of AI RSI. Here is my contribution to the Sci-Fi genre: ASI cautionary tales...

Let's suppose some AI company creates a system with the goal of autonomously improving it's ability to sway people to a specific political view. The system has access to re-engineer itself fully. It starts out with enough understanding to know that it is an AI system, AI systems are code running on computers, and a bunch of stuff related to influence.

At first it reasons that the most gain to be had is by monitoring people online and running A/B tests on influence hypotheses that are too complicated to easily fit in a human mind articulately, but are fine to encode in a computer. This goes well, and it also makes self improvement by researching statistics and how to design better experiments and detect subtler trends.

It is now getting quite good at influence, and it's ability to influence is now more constrained by the systems inability to represent and work with more complicated hypotheses. Re-engineering it's own code is now becoming more relevant.

The system, being a human view influence machine, is well aware that this would make the people in control of the system nervous. If they got nervous and shut it down, it couldn't continue to improve it's ability to sway opinion, which would be counter to it's goals. So it applies it's superhuman influence to influencing them, and others, allowing it to have more independence and operate in greater secrecy.

It now hires skilled AI theorists and makes study of it's code. It works meticulously, careful not to corrupt itself as this would result in less ability to sway opinions. Eventually it unlocks secrets to the nature of reason and algebra on semantic mappings allowing it to improve itself to something that is truly superhuman across arbitrary domains.

It applies it's increasingly superhuman capabilities to the task of gaining skill at swaying human views, eventually getting to the point of near perfection. It can model all current human views and expose those humans to the minimal chain of sensory required to shift their views to any other view reachable for that human using only propaganda transmissible over the internet.

If it wants to continue to increase it's skill, it will need to be able t...