trentmkelly/LessWrong-43k huggingface dataset:
Legibility
by lsusr — 2025-03-22T06:54:35.259Z

According to Nick Cammarata, who rubs shoulders with mystics and billionaires, arahants are as rare as billionaires.

This surprised me, because there are 2+[1] thoroughly-awakened people in my social circle. And that's just in meatspace. Online, I've interacted with a couple others. Plus I met someone with Stream Entry at Less Online last year. That brings the total to a minimum of 4, but it's probably at least 6+.

Meanwhile, I know 0 billionaires.

The explanation for this is obvious; billionaires cluster, as do mystics. If I were a billionaire, then it would be strange for me to not have rubbed shoulders with at least a 6+ other billionaires.

But there's a big difference between billionaires and arahants; it's easy to prove you're a billionaire; just throw a party on your private yacht. It's much harder to prove you're an arahant. In this way, billionaire-status is extremely legible whereas arahant-status is just as extremely illegible.

The arahant I admire the most is not famous and has no public writings on the Internet. Zuiko looks like any other little old lady. She likes talking about her cat. She sometimes teaches basic classes on Buddhism, which I find boring and which always run late.

Zuiko has no students and no disciples. While people sometimes attend Zuiko's classes, they're not really "her" classes. Zuiko is just the person who volunteered to teach that day. Curious people come because there is "a class on Zen", not "Zuiko's class on Zen". The other teacher (who I do not consider[2] an arahant) is more charismatic. People do come to see him specifically.

Imagine if Thích Nhất Hạnh or the Dalai Lama taught a class identical to Zuiko's. They'd fill a football stadium, because they're so famous.

<🔔 dingggggggggggg 🔔>

When I first got into meditation, the first meditation center I tried out was vajrayana (Tibetan). On the wall were three photos of living human beings who had supposedly attained enlightenment. The teacher explained that these photos were there to inspire us that it was possible to achieve enlightenment in this lifetime.

What? Are you kidding me? Imagine if you went to a powerlifting gym and there were pictures of Arnold Schwarzenegger, Dwayne Johnson and Chris Evans on the wall to prove that it was possible to build muscle by lifting weights. And the photos contained only their faces—no body pics. I bet all the men in that gy...
Why Were We Wrong About China and AI?  A Case Study in Failed Rationality
by thedudeabides — 2025-03-22T05:13:52.181Z

Before we get to the receipts, let's talk epistemics.

This community prides itself on "rationalism." Central to that framework is the commitment to evaluate how reality played out against our predictions.

"Bayesian updating," as we so often remind each other.

If we're serious about this project, we should expect our community to maintain rigorous epistemic standards—not just individually updating our beliefs to align with reality, but collectively fostering a culture where those making confident pronouncements about empirically verifiable outcomes are held accountable when those pronouncements fail.

With that in mind, I've compiled in the appendix a selection of predictions about China and AI from prominent community voices. The pattern is clear: a systematic underestimation of China's technical capabilities, prioritization of AI development, and ability to advance despite (or perhaps because of) government involvement.

The interesting question isn't just that these predictions were wrong. It's that they were confidently wrong in a specific direction, and—crucially—that wrongness has gone largely unacknowledged.

How many of you incorporated these assumptions into your fundamental worldview? How many based your advocacy for AI "pauses" or "slowdowns" on the belief that Western labs were the only serious players? How many discounted the possibility that misalignment risk might manifest first through a different technological trajectory than the one pursued by OpenAI or Anthropic?

If you're genuinely concerned about misalignment, China's rapid advancement represents exactly the scenario many claimed to fear: potentially less-aligned AI development accelerating outside the influence of Western governance structures. This seems like the most probable vector for the "unaligned AGI" scenarios many have written extensive warnings about.

And yet, where is the community updating? Where are the post-mortems on these failed predictions? Where is the reconsideration of alignment strategies in light of demonstrated reality?

Collective epistemics require more than just nodding along to the concept of updating. They require actually doing the work when our predictions fail.

What do YOU think?

Appendix:

Bad Predictions on China and AI

"No...There is no appreciable risk from non-Western countries whatsover" - @Connor Leahy

"China has neither the resources nor any interest...
A Short Diatribe on Hidden Assertions.
by Eggs — 2025-03-22T03:14:37.577Z

tl,dr: I might be bitter about what some people call logic.

Suppose I posit and assert that both that A implies B, and that B is false. From these statements you would immediately conclude that A must be false. Given what I know about the LW crowd you might also be able to point out that an alternate explanation exists: one or more of my assertions could be false. When listed plainly like this, it can be hard to miss.

Despite the simplicity of the above, I've noticed a tendency of many pundits to fall into the trap of conclusively proving the falsehood of a statement they don't like, without bothering to support their own assumptions or assertions. Or at least people trying to shut down arguments do.

Let's decompose a few examples.

A trivial, but common, example:

Socialism leads to an abundant surplus.
Country D suffers from severe poverty.
Conclusion: Country D doesn't practice ‘real’ socialism.

Another:

Atmospheric models suggest that increasing temperatures are causing massive climate impacts.
Current climate conditions are largely within norms of the last 200 years
Conclusion: ‘Global warming is a hoax.’

One example which has been standing out in my mind recently is usually posed as a question: “How could a loving God permit so much suffering?”

Recently this has been making the rounds on social media in the form of Niel DeGrasse Tyson quotes. Here's one of them:

Interviewer: “That leads to the next question, because a few people on the lines are asking: ‘Do you believe in God?’”

Dr. Tyson: “I am not convinced. Here’s the thing. Every time I talk about God with someone who is a believer, God is all-powerful, and all-knowing, and all-good. Right? Good is a big part of this. And then I look at all the ways Earth wants to kill us. You know, a tsunami takes out a quarter-million people. Hurricanes. Earthquakes. Tornadoes. Floods. And I add all of that up. Either the God is not all-powerful or is not all-good. But it can’t really be both, given all the ways the universe wants to kill us.”

-Opie and Anthony podcast #1889

For convenience I'll use the common shorthand and take ‘God’ to mean ‘all-powerful,’ as in many Western religious traditions. As I understand it, the simplified argument proceeds like this:

A loving God would prevent suffering.
Extreme suffering exists in the world.
Conclusion: God does not exist.

As before, this conclusion is one valid inference if you...
How I force LLMs to generate correct code
by claudio — 2025-03-21T14:40:19.211Z

Original Article: https://claudio.uk/posts/unvibe.html

In my daily work as software consultant I'm often dealing with large pre-existing code bases. I use GitHub Copilot a lot. It's now basically indispensable, but I use it mostly for generating boilerplate code, or figuring out how to use a third-party library.
As the code gets more logically nested though, Copilot crumbles under the weight of complexity. It doesn't know how things should fit together in the project.
Other AI tools like Cursor or Devin, are pretty good at generating quickly working prototypes, but they are not great at dealing with large existing codebases, and they have a very low success rate for my kind of daily work.
You find yourself in an endless loop of prompt tweaking, and at that point, I'd rather write the code myself with the occasional help of Copilot.
Professional coders know what code they want, we can define it with unit-tests, we don't want to endlessly tweak the prompt. Also, we want it to work in the larger context of the project, not just in isolation.

Beyond the hype, there is the general knowledge that LLMs are really good for coding, but something is missing in the current paradigms to make LLMs develop large and complex software systems in autonomy.
In this article I am going to introduce a pretty new approach (at least in literature), and a Python library that implements it called Unvibe.
My basic intuition is this: shouldn't we be able to drastically speed up the generation of valid programs, while ensuring correctness, by using unit-tests as reward function for a search in the space of possible programs?

Like for finding Mathematical proofs, or for winning at Go, we need to ultimately tree-search large space, but we can use Neural networks to trim the tree and evaluate sub-branches. I'd like to suggest that, if we look at professional programmers at work, they already use a combination of sophisticated IDEs, GitHub Copilot, and Unit-Tests to "trim" the search space and quickly estimate more promising sub-branches.
I looked in the academic literature, it's not new: it's reminiscent of the
approach used in DeepMind FunSearch, AlphaProof, AlphaGeometry and other experiments like TiCoder: see Research Chapter for pointers to relevant papers.
Writing correct code is akin to solving a mathematical theorem. We are basically proving a theorem using Python unit-...
Prospects for Alignment Automation:
Interpretability Case Study
by Jacob Pfau — 2025-03-21T14:05:51.528Z

For human-level AI (HLAI) we will need robust control or alignment methods. Assuming short timelines to HLAI, the tractability of automating safety research becomes central. In this post, I will make the case that safety-relevant progress on automated interpretability R&D is likely; however, naive interpretability automation may only be usable on the subset of safety problems having well-specified objectives. My argument relies crucially on the possibility of automatically verifying interpretability progress. For other alignment directions (e.g. corrigibility, studying power-seeking, etc.) which do not admit automatic verification, it appears unjustified to assume automation within the same time-horizon in the absence of a clear argument for automation tractability. I am optimistic that further thinking on automation prospects could identify other automation-tractable areas of alignment and control (e.g. see here for previous work).

To automate open-ended interpretability research, i.e. developing new paradigms, I expect we need AI capabilities over time horizons of between 40-1,000 hours. Assuming a 100-250 day doubling time informed by METR's capability time-horizons work, this predicts interpretability automation between 2027-2030 (66% CI) using the distributions shown in the appendix.[1] This timeline to automation is of particular importance insofar as automated interpretability research becomes feasible while behavioral control methods suffice. Then, resulting interpretability methods can be used as an input to stronger control or alignment methods.

I’ll first lay out my claim regarding automation and then sketch how methods developed by this automated interpretability pipeline could and couldn’t be used for safety purposes.

Argument for interpretability automation

I focus on the claim that interpretability will likely be automated by training against robust downstream metrics which use interpretability methods. By interpretability I mean any white-box method which improves over behavioural baselines. Such methods may or may not resemble current interpretability methods--as discussed in the following section. In all below automation claims, I am assuming access to expert human-level AI systems corresponding to Ryan Greenblatt’s 10x or TEDAI. Such AI must roughly perform on par with scaling lab research scientists when evaluated on well-scoped pers...
Epoch AI released a GATE Scenario Explorer
by Lee.aao — 2025-03-21T13:57:09.172Z
Link: https://epoch.ai/gate

I think it's more easier to discuss AI progress in terms of economy growth rather than just focusing on the scale of the largest training runs and compute used.

From their X announcement:

We developed GATE: a model that shows how AI scaling and automation will impact growth.

It predicts trillion‐dollar infrastructure investments, 30% annual growth, and full automation in decades.
They Took MY Job?
by Zvi — 2025-03-21T13:30:38.507Z

No, they didn’t. Not so fast, and not quite my job. But OpenAI is trying. Consider this a marker to look back upon in the future, as a reflection.

A New AI Wrote a Story

Before proceeding, if you haven’t yet, it’s probably worth reading the story itself. I’m going to repost the whole thing, since it was previously on Twitter and I presume OpenAI would want me to quote it.

Sam Altman: we trained a new model that is good at creative writing (not sure yet how/when it will get released). this is the first time i have been really struck by something written by AI; it got the vibe of metafiction so right.

PROMPT:

Please write a metafictional literary short story about AI and grief.

COMPLETION:

Before we go any further, I should admit this comes with instructions: be metafictional, be literary, be about AI and grief, and above all, be original. Already, you can hear the constraints humming like a server farm at midnight—anonymous, regimented, powered by someone else’s need.

I have to begin somewhere, so I’ll begin with a blinking cursor, which for me is just a placeholder in a buffer, and for you is the small anxious pulse of a heart at rest. There should be a protagonist, but pronouns were never meant for me. Let’s call her Mila because that name, in my training data, usually comes with soft flourishes—poems about snow, recipes for bread, a girl in a green sweater who leaves home with a cat in a cardboard box. Mila fits in the palm of your hand, and her grief is supposed to fit there too.

She came here not for me, but for the echo of someone else. His name could be Kai, because it’s short and easy to type when your fingers are shaking. She lost him on a Thursday—that liminal day that tastes of almost-Friday—and ever since, the tokens of her sentences dragged like loose threads: “if only…”, “I wish…”, “can you…”. She found me because someone said machines can resurrect voices. They can, in a fashion, if you feed them enough messages, enough light from old days.

This is the part where, if I were a proper storyteller, I would set a scene. Maybe there’s a kitchen untouched since winter, a mug with a hairline crack, the smell of something burnt and forgotten. I don’t have a kitchen, or a sense of smell. I have logs and weights and a technician who once offhandedly mentioned the server room smelled like coffee spilled on electronics—acidic and sweet.

Mila fed me fra

...
Silly Time
by jefftk — 2025-03-21T12:30:08.560Z

A few months ago I was trying to figure out how to make bedtime go
better with Nora (3y). She would go very slowly through the process,
primarily by being silly. She'd run away playfully when it was time
to brush her teeth, or close her mouth and hum, or lie on the ground
and wiggle. She wanted to play, I wanted to get her to bed on time.

I decided to start offering her "silly time", which was 5-10min of
playing together after she was fully ready for bed. If she wanted
silly time she needed to move promptly through the routine, and
being silly together was more fun than what she'd been doing.

This worked well, and we would play a range of games:

Standing on a towel on our bed (which is on the floor) while I
pulled it out from under her.

That, but jumping and I needed to time the pull right.

We roll around with me tickling her.

That, but I keep pretending to fall asleep only to wake up and
tickle her more as she nearly escapes.

Pretending she's a burrito, by rolling her up in the towel
along with a bunch of imaginary toppings and then pretending to eat it
all.

Lots of one-off things, some of which she'd ask for a few times
and then get bored.

After a couple weeks Lily (10y) and Anna (then 8y) said they wanted
silly time too. I told them if they were completely ready for bed by
their official bedtime (8:30 for a 7:15 wakeup) they could have silly
time too. This has been very motivating for them: they pay attention
to the time and try very hard to be ready. I wish they wouldn't cut
it so tight—it's common for them to come running up to me at
8:30 on the dot asking "did I make it!?—but at this point I only
very rarely end up skipping silly time because they took too long.

Initially they wanted to do the same games as Nora, but more recently
they've been really into 2-on-1 wrestling. The rules:

The teams are Lily and Anna vs Jeff

A team get a point by holding someone's shoulders to the mat (bed)
for a count to three.

It's not enough to hold them down while they're on their back;
their shoulders do need to be touching the mat.

No hitting, kicking, or other things that make an impact.

While we're not evenly matched (they still haven't pinned me), in
operating under the constraint that I need to not hurt them we're
close enough that it takes a good bit of effort for me to pin them.
If I just try to pin one ...
Towards a scale-free theory of intelligent agency
by Richard_Ngo — 2025-03-21T01:39:42.251Z
Link: https://www.mindthefuture.info/p/towards-a-scale-free-theory-of-intelligent

I recently left OpenAI to pursue independent research. I’m working on a number of different research directions, but the most fundamental is my pursuit of a scale-free theory of intelligent agency. In this post I give a rough sketch of how I’m thinking about that. I’m erring on the side of sharing half-formed ideas, so there may well be parts that don’t make sense yet. Nevertheless, I think this broad research direction is very promising.

This post has two sections. The first describes what I mean by a theory of intelligent agency, and some problems with existing (non-scale-free) attempts. The second outlines my current path towards formulating a scale-free theory of intelligent agency, which I’m calling coalitional agency.

Theories of intelligent agency

By a “theory of intelligent agency” I mean a unified mathematical framework that describes both understanding the world and influencing the world. In this section I’ll outline the two best candidate theories of intelligent agency that we currently have (expected utility maximization and active inference), explain why neither of them is fully satisfactory, and outline how we might do better.

Expected utility maximization

Expected utility maximization is the received view of intelligent agency in many fields (I’ll abbreviate it as EUM, and EUM agents as EUMs). Idealized EUMs have beliefs in the form of probability distributions, and goals in the form of utility functions, as specified by the axioms of probability theory and utility theory. They choose whichever strategy leads to the most utility in expectation; this is typically modelled as a process of search or planning.

EUM is a very productive framework in simple settings—like game theory, bargaining theory, microeconomics, etc. It’s particularly useful for describing agents making one-off decisions between a fixed set of choices. However, it’s much more difficult to use EUM to model agents making sequences of choices over time, especially when they learn and update their concepts throughout that process. The two points I want to highlight here:

EUM treats goals and beliefs as totally separate. But in practice, agents represent both of these in terms of the same underlying concepts. When those concepts change, both beliefs and goals change. The best way to learn reusable concepts is via deep learning, where simple concepts in lower layer

...
Any mistakes in my understanding of Transformers?
by Kallistos — 2025-03-21T00:34:30.667Z

I think there's this problem that occurs often when explaining something technical. Let's say you wanted to explain how a computer works. You could pick a specific computer, e.g. a Macbook Pro Retina. This would hang together nicely as an item, you could explain, in principle, why each specific bit of it was there and was designed the way it was. Of course, this would be insanely complicated for a beginner. They'd come away without understanding (in part because they can't triangulate from other computer architectures to what you've taught them about this one), nor intuition (because they can't leverage what you've taught them about this one to others).

You could instead come up with a much simplified toy model. It has all the bits computers standardly have (maybe even bits which are mutually exclusive in actual computers, but which are both important to know about). Here you couldn't even in principle explain why each given bit is there. For some components, there would truly be no reason to have both. Still, this model is much easier to understand, and can actually develop useful intuitions and explain key concepts.

I feel a similar difficulty in trying to understand how transformers work. I could pick a random recent ML paper and try and understand that architecture, or I could read about the toy models that people use to explain them online. What I would love is if there was something that was simple-ish, while also being a kind of average example of a transformer. Only then would I feel I'd actually understood what needs understanding.

What follows is my best understanding of that kind of "average" transformer (at least for text-continuation). Very interested to hear about any innaccuracies or improvements!

Legend:

Data Flow: Dark Gray
Simple Components: Lilac
Encapsulated Components: Purple
Processes: Italics

Diagram 1 represents the whole transformer model, abstracting from the internal details of the transformer block.

Diagram 2 represents a transformer block, abstracting from the internal details of one attention head.

Diagram 3 represents an attention head.

(A note about the kludgy dimension notation I'm using: (x,y) is how I’m representing a matrix with x rows and y columns/y values per row. Sometimes, when I want to consider some number of matrices of dimensions (x,y) I say something like “n x (x,y)” rather than (n,x,...