trentmkelly/LessWrong-43k huggingface dataset:
Understanding AI World Models w/ Chris Canal
by jacobhaimes — 2025-01-27T16:32:47.724Z
Link: https://kairos.fm/muckraikers/e011/

On our latest

episode

of

muckrAIkers

we are joined by our first ever guest, Chris Canal!

You can find the show on Spotify, Apple Podcasts, YouTube, or wherever else you listen.

Chris is the co-founder of

EquiStamp

, a third-party evaluator providing objective and accurate evaluations of frontier LLMs and the systems built off of them. The group has been a key contractor for METR, providing baselines for their recent work

RE-Bench

, which evaluates frontier LLM capabilities in AI R&D.

In this ~3 hour long episode on understanding AI world models, we get into test-time compute, the moving goalposts of “AGI,” and do a deep dive on the orthogonality thesis.

EquiStamp is hiring, so if you're a software developer interested in a fully remote opportunity with flexible working hours, join the EquiStamp

Discord server

and message Chris directly; oh, and let him know muckrAIkers sent you!

As always, all sources referenced throughout the episode can be found on the showpage at

https://kairos.fm/muckraikers/e011

Given the imprecise sensationalism that permeates AI discourse, it is important to have a critical perspective when reading press releases, news coverage, and even academic papers about AI. On muckrAIkers, Dr. Igor Krawczuk and I (Jacob Haimes) contextualize these developments, separating muck from meaning, allowing all of us to better understand the motivations and likely next steps of major players in the AI space.

muckrAIkers is a

Kairos.fm

production
The present perfect tense is ruining your life
by PatrickDFarley — 2025-01-27T16:14:48.843Z

Yes I’m talking about grammar. The present perfect tense describes actions that happened in the past but are relevant in the present. You say, “I have seen that movie,” meaning you saw it in the past, but what’s important is that presently you are a person who saw it.

How is a verb tense ruining my life?

There’s a world of difference between wanting to do something and wanting to have done something.

I’m not the first to find this distinction interesting:

I don’t believe any of you have ever read PARADISE LOST, and you don’t want to. That’s something that you just want to take on trust. It’s a classic, just as Professor Winchester says, and it meets his definition of a classic — something that everybody wants to have read and nobody wants to read.

Mark Twain, “The Disappearance of Literature”

What’s dangerous is when we conflate the two tenses. We say, “I want to do X,” when really we want to have done X. I’ve often caught myself doing this, and I see others doing it too. It’s a form of deception (usually self-deception), and it sets us up for failure and frustration in our efforts.

Let’s talk about the nature of this mistake, why we make it, and how it comes back to bite us.

so fun

What does it mean to want in the present perfect tense?

Let’s look closely at the difference between wanting to read Paradise Lost and wanting to have read Paradise Lost.

Wanting in the present tense is simple: when you want to do something, your desire is for the act itself.

Wanting in the present perfect tense is tricky: when you want to have done something, your desire is for a future state where you already did the thing and acquired the memories/experience of it.

Notice the language there—you acquired something, like you’re doing work and getting a paycheck at the end. In fact even the original wording implies this: in English, we use the word “have” both to mean possession and to construct the present perfect tense (“I have a car” / “I have been to Florida”). That’s not a coincidence! You can see how statements in the present perfect tense could also be interpreted as statements about possession: saying I have been to Florida is basically the same as saying I have (i.e. possess) the experience/memory of being in Florida. This double usage of the verb “to have” goes all the way back t...
The Upcoming PEPFAR Cut Will Kill Millions, Many of Them Children

by Bentham's Bulldog — 2025-01-27T16:03:51.214Z

Crossposted from my blog.

(This could end up being the most important thing I’ve ever written. Please like and restack it—if you have a big blog, please write about it).

A mother holds her sick baby to her chest. She knows he doesn’t have long to live. She hears him coughing—those body-wracking coughs—that expel mucus and phlegm, leaving him desperately gasping for air. He is just a few months old. And yet that’s how old he will be when he dies.

The aforementioned scene is likely to become increasingly common in the coming years. Fortunately, there is still hope.

Trump recently signed an executive order shutting off almost all foreign aid. Most terrifyingly, this included shutting off the PEPFAR program—the single most successful foreign aid program in my lifetime. PEPFAR provides treatment and prevention of HIV and AIDS—it has saved about

25 million people

since its implementation in 2001, despite only taking less than 0.1% of the federal budget. Every

single day that it is operative

, PEPFAR supports:

More than 222,000 people on treatment in the program collecting ARVs to stay healthy;
More than 224,000 HIV tests, newly diagnosing 4,374 people with HIV – 10% of whom are pregnant women attending antenatal clinic visits;
Services for 17,695 orphans and vulnerable children impacted by HIV;
7,163 cervical cancer screenings, newly diagnosing 363 women with cervical cancer or pre-cancerous lesions, and treating 324 women with positive cervical cancer results;
Care and support for 3,618 women experiencing gender-based violence, including 779 women who experienced sexual violence.

The most important thing PEPFAR does is provide life-saving anti-retroviral treatments to

millions of victims of HIV

. More than

20 million people living

with HIV globally depend on daily anti-retrovirals, including over half a million children. These children, facing a deadly illness in desperately poor countries, are now going to be left on their own, and a lot of them are going to die.

Imagine a baby you care about having to face that. Imagine them having a horrible disease, a disease that will likely kill them if untreated. But they have been getting treatment—treatment that makes it so that they will not die from the disease but be able to grow and flourish. Now, with the stroke of a pen, their medicine will be cut off—probably they will die in a few months.

Kelsey Piper writes

:

There are

...
Is the output of the softmax in a single transformer attention head usually winner-takes-all?
by Linda Linsefors — 2025-01-27T15:33:28.992Z

Using the notation from here: A Mathematical Framework for Transformer Circuits

The attention pattern for a single attention head is determined by A=softmax(xTWTQWKx), where softmax is computed for each row of xTWTQWKx.

Each row of A gives the attention pattern for the current token. Are these rows (post softmax) typically close to one-hot? I.e. are they mainly dominated by a single attention (per current token).

I'm interested in knowing this for various types of transformers, but mainly for LLM and/or frontier models.

I'm asking because I think this has implication for computations in super-position.
To know or not to know
by arisAlexis — 2025-01-27T13:17:33.672Z

We are on the brink of the unimaginable. Humanity is about to cross a threshold that will redefine life as we know it: the creation of intelligence surpassing our own. This is not science fiction—it’s unfolding right now, within our lifetimes. The ripple effects of this seismic event will alter every aspect of society, culture, and existence itself, faster than most can comprehend.

But here’s the pressing question: should everyone know about this impending transformation, or are some truths too overwhelming to bear?

In this article, I won’t debate the feasibility of Artificial General Intelligence (AGI) or its exponential progress. Instead, I’ll dive into the heart of the matter: the arguments for and against widespread awareness of these changes. Should humanity be fully informed and mentally prepared for a future dominated by superintelligence, or is ignorance a better path to preserve our sanity and way of life? Let’s explore.

I’ll go with the arguments that favor knowledge first. Even if you can’t actually influence the outcome (although it can be debated till death, there could be a miniscule effect of the populace affecting big tech and government policies but let’s assume they can’t), you can certainly affect other aspects of your personal life such as the following.

Live fast

Living life to the fullest in the scene of the roaring 20s or the cocktail parties in the bunkers while Hitler was bombing London. I am recently being moved a lot by conversations with Ukranians that continue to try to live a normal life in Kyiv. There is an awesome aspect of human psychology that changes gears to live fast and die young mentality. It can cut both ways, I am certainly not fascinated by doing stupid stuff like snowballing drugs to end up like Heath Ledger but taking life changing trips now instead of later. Telling the girl/boy you have a crush today and generally forfeiting the work hard to satisfy the needs of my pension or my unborn grandchildren mentality that boomers grew up with. There is no pension if you are a young middle aged adult now. Your boss will probably be unemployed soon so don’t worry about your promotion coming in 5 years that you need to work overtime today.

Catch the longevity escape velocity train

For me this is one of the most important if not the most important aspect of knowing. It comes in stark contrast of the first argument bu...
My supervillain origin story
by Dmitry Vaintrob — 2025-01-27T12:20:46.101Z

When I started graduate school (for math), I was very interested in big ideas. I had had a couple experiences of having general research intuitions pan out really well and felt like the core of good research is having a brave idea, a

gestalt

. I went into grad school looking for the “gestalt people”. The people whose math had that mysterious, cutting edge flavor but was not too pop (at the time the sexiest thing around was higher category theory and I was drawn to it and tried to learn it, but I didn’t want to do the “common” thing of working in that field). I ended up choosing an advisor, and skipped over any computational or “applications-driven” (insofar as doing calculations in Galois theory or string theory counts as applications) stuff that he recommended working on. It really wasn’t my thing: you had to read long technical papers, use the right lemmas, apply them to get actual numbers (yuck) that people would then build upon in future research. I wanted to have the big ideas.

I ended up coming up with my own research project to check that an object my advisor had discovered – a certain new category associated to a Lie group – is equivalent to another category that I defined by applying higher category theory to some combinatorial data. This was exactly the kind of general thing I wanted to have: no numbers, purely theoretical, applies to a fully general class of groups: “Gestalt”. My advisor was skeptical but let me work on it.

I ended up trying to prove a false result for a year and a half.

It wasn’t one of these “cute” false results, where all the cases you check work but there’s an unexpected edge case or counterexample. It was a straight-up false result where, if you really carefully worked out the smallest nontrivial example, you would see it’s false. The problem is that the falseness was still a little sneaky. The thing is that in this flavor of representation theory the smallest interesting case, the group GL2 (i.e. the group of 2×2 matrices) is already somewhat hard (there is an infamous

300-page book

about it), and while you wouldn’t have to read 300 pages to find the contradiction, you would have to intentionally work on it: clearly think through the implications of the result being correct with a view towards looking for a concrete computation or check that would lead to a contradiction. Instead of this, I alternated betwe...
The Clueless Sniper and the Principle of Indifference
by Jim Buhler — 2025-01-27T11:52:57.978Z

You are one of the best long-range snipers in the World. You often have to eliminate targets standing one or two miles away and rarely miss. Crucially, this is not because you are better than others at aligning the scope of your rifle with the target’s head before taking the shot. Rather, this is because you are better at accounting for the external factors that will influence the bullet’s direction (e.g., the wind, gravity, etc.). Estimating these external factors is crucial for your success. One day, you are on a roof with binoculars, looking at a building four miles away that you and your team know is filled with terrorists and their innocent children. Your superior is beside you, looking in the same direction. Your mission is to inform your allies closer to the building if you see any movement. After a long wait, the terrorist leader comes out, holding a child close to him. He knows ennemies might be targetting him and would not want to hurt the kid. They are hastily heading towards another building nearby. You grab your radio to inform your allies but your superior stops you and hands you your sniper. “He’s so exposed. This is a golden opportunity for you.” she says. “I reckon we’ve got two minutes before they reach the other building. Do you think you can get him?“. You know that your superior generally accepts risking the lives of innocents but only if there are more bad guys than innocents killed, in expectation. “Absolutely not.” you respond. “We are four miles away! I’ve never taken a shot from this far. NO ONE has ever hit a target from this far. I’m just as likely to hit the kid.” Your superior takes a few seconds to think. “You always say that where the bullet ends up is the result of an equation: ‘where you aim + the external factors’, right?” she says. “Yes,” you reply nervously “And I have no idea how to account for the external factors from that far. There are so many different wind layers between us and the target. The Earth’s rotation and the spin drift of the bullet matter also a hell of a lot from this distance. I would also have to factor in the altitude, humidity, and temperature of the air. I don’t have the faintest idea what this sums up to overall from this far! I’m clueless. I wouldn’t be more likely to get him and not the kid if I were to shoot with my eyes closed.” Your superior turns her head towards you. “Principle of ind

...
Scanless Whole Brain Emulation
by Knight Lee — 2025-01-27T10:00:08.036Z

Warning: this is very speculative and I have no expertise.

If we can build superintelligence by starting with simulated humans, it should be safer.

Scanless Whole Brain Emulation attempts to simulate the development of the nervous system from the very start, thus not requiring a brain scan.

Hard, but necessary anyways?

For a Whole Brain Emulation to be useful, the simulated humans need to be able to learn. They need to understand new things and build on top of their new understandings, otherwise they would be intellectually disabled.

They need some level of neuroplasticity. The system needs to simulate how neurons use past information, as well how neurons form and prune connections. It needs to simulate the part of brain development in adulthood.

Given that it must simulate the part of brain development in adulthood, simulating brain development from the very start might not be that much harder.

But how would you even start?

The idea is to create protein models using chemistry simulations, and then create organelle models from protein simulations, and then create cell models from organelle simulations, and finally create "Scanless Whole Brain Emulation" from cell simulations.

Each model is created by running simulations of lower level mechanics, and fitting the simulation to empirical data. Once the simulation matches empirical data well enough, a model is then trained to match the simulation's behaviour.[1]

This differs from traditional Whole Brain Emulation, where the neuron models come from purely empirical data. Gaining empirical data about which neuron connects to which other neuron requires a brain scan, but brain scans are very hard to do.

By using lower level simulations, we can make a far finer model of the neuron, which captures all the important behaviours and computations that a neuron does.[2]

A human cell such as a neuron has 750MB of data in its DNA or "source code", and may store a smaller amount of data in its epigenetics or "memory." It may store a much smaller amount of data in the concentrations of various molecules and ions, which is the very short term memory or "CPU buffer."

Its behaviour is very complex in the sense it is hard to understand. But it isn't very "computationally complex" in the sense the nonrandom and important computations are fewer than the computations in a cheap computer.

If we can model human neurons and cells accurately enough t...
Death vs. Suffering: The Endurist-Serenist Divide on Life’s Worst Fate


by Alex_Steiner — 2025-01-27T03:59:40.279Z

Author’s Note
Longtime LW lurker, occasional contributor (under other aliases). This post introduces a taxonomy of preferences centered on one question: What do agents treat as the "worst thing"?

Core Framework:

Endurists: Death as the ultimate evil (life at all costs).
Serenists: Suffering as the ultimate evil (non-existence over agony).

This framework is written in LW’s analytical style, but I’ll state upfront: I lean Serenist. My goal isn’t neutrality—it’s to see if explicit value categories can sharpen debates about medical ethics, AI alignment, and institutional design.

Cross-posted from Qualia Advocate.

Consider the following question: What is the worst thing that could happen to a person?
I would claim there are two main answers: to die, or to exist in a state of extreme suffering—to be in hell.

“Better to suffer than to die.”
— Jean de La Fontaine

Endurists: Those who see death as the worst outcome. They would choose a life of pain over non-existence.

“For it would be better to die once and for all than to suffer pain for all one's life.”
― Aeschylus, Prometheus Bound

Serenists: Those who see suffering as the worst outcome. They believe some lives aren’t worth living, and non-existence can be better than extreme suffering.

This divide goes beyond personal choice. It shapes religions, philosophy, medical ethics, and laws. It affects how societies treat their members, with Endurist values often being forced on those who hold Serenist views. The purpose of this post is to explore these two views.

Psychological Profiles

Endurists and Serenists differ fundamentally in how they process and value suffering versus existence.

A Twitter poll by Spencer Greenberg captured this divide by suggesting the following scenario:

When asked about enduring terrible constant pain for 11 months with a 1% survival rate, 40.3% would "fight till the end" (Endurists), while 59.7% would "seek euthanasia" (Serenists).

The split might sharpen further if posed with zero chance of survival—a pure Endurist would still choose those months of suffering just to exist longer.

Endurists often psychologically minimize suffering’s importance or treat it as less "real" than existence itself. This helps explain their choices: if you don’t fully acknowledge suffering’s weight, existence naturally takes priority.

Consider these contrasting responses to suffering:

In the Soviet Gulag of Kolyma, as documen...
Are we trying to figure out if AI is conscious?
by Kristaps Zilgalvis — 2025-01-27T01:05:07.001Z
Link: https://open.substack.com/pub/kristapszilgalvis/p/are-we-trying-to-figure-out-if-ai?r=27orkx&utm_campaign=post&utm_medium=web&showWelcomeOnShare=false

My aim in this article is to examine the field of consciousness research with an analysis of 78 of the most important papers in the field. In doing so, I compare the empirical foundations these disciplines rely upon and detail how each approaches the measurement of conscious experience.
Lazy Hasselback Pommes Anna
by Brendan Long — 2025-01-26T21:30:36.587Z
Link: https://www.brendanlong.com/lazy-hasselback-pommes-anna.html

Do you have an insatiable hunger for potatoes? Do you have a fully-sated hunger for complicated recipes and having to cook all the time? This version of Pommes Anna, simplified and altered to an extent that will make French people cry, might be the recipe for you!

I try to eat unreasonable amounts of potatoes every day, so I want a food that doesn't take a lot of time to make, can be stored and reheated, and remains palatable after eating it for every meal for months at a time. The advantages of this recipe are:

There's only 3 (required) ingredients.
The crispy texture on top makes it more palatable long-term than mashed potatoes.
The result can be reheated in a microwave in 2 minutes.

The main downsides are:

It takes around an hour and a half to make (15-30 minutes of prep, 50 minutes of baking).
You absolutely need a mandoline. Do not try to thinly slice 5 lbs of potatoes by hand.

Recipe

Prep Time: 30 minutes

Cook Time: 70 minutes

Servings: 6

Nutrition[1]:

2493 calories total (312 calories per serving)
48 g protein (8 g per serving)
49 g fat (8 g per serving)
478 g carbs (80 g per serving)
Provides adequate amounts of most vitamins and minerals except for B12, Vitamin D, Vitamin E, Calcium, and Selenium.

If this is your whole diet, you may want to eat some eggs along with it.

Equipment

9x13 inch casserole dish
Mandoline

Safety Note: Always use mandoline guard!

Ingredients

5 lbs Yukon Gold potatoes
1 tbsp table salt
1/2 stick (1/4 cup) butter[2]
Additional seasonings like garlic, pepper or thyme (optional)

Note: Recipe can be scaled to any baking dish. Fill dish with peeled potatoes to measure, then scale butter and salt proportionally.

Directions

1. Pre-heat oven to 400° F.

2. Peel the potatoes and then slice them thinly with a mandoline (I use the 2mm setting on mine). Cut any leftover potato pieces into thin strips. Add all of the potato to a very large bowl.

3. Melt the butter.

4. Add the butter, salt and any additional seasonings to the potato slices and mix it in.

5. Add mixture to the casserole dish. Placing the slices edge-up or randomly (my preference) will give you the most varied and crispy texture. Placing the slices flat will give you a more consistent custardy interior. Pour any additional liquid at the bottom of your bowl over the top.

6. Bake uncovered for 50 minutes.

7. Let cool and then slice and eat, or cool entirely in the fridge for solid pretty-looking slices that rehea...