trentmkelly/LessWrong-43k huggingface dataset:
3. Why impartial altruists should suspend judgment under unawareness
by Anthony DiGiovanni — 2025-06-08T15:06:30.594Z

To recap, first, we face an epistemic challenge beyond uncertainty over possible futures. Due to unawareness, we can’t conceive of many relevant futures in the first place, which makes the standard EV framework ill-suited for impartial altruistic decision-making. And second, we can’t trust that our intuitive comparisons of strategies’ overall consequences price in factors we’re unaware of with enough precision. We’ll need to evaluate these consequences with a framework that explicitly accounts for unawareness, that is, unawareness-inclusive expected value (UEV).

Here, I’ll more specifically unpack the UEV model, and argue that we can’t compare the UEV of any given strategy with another. If so, we don’t have a reason to choose one strategy over another based purely on the impartial good. We should, instead, suspend judgment as far as impartial altruism is concerned. I’ll conclude by illustrating this problem with a worked example, building on our case study from before. (My arguments here aren’t meant to refute the specific approaches EAs have proposed for comparing strategies under unawareness. That’s for the final post.)

Unawareness-inclusive expected value (UEV)[1]

Key takeaway

To get the imprecise “EV” of a strategy under unawareness, we take the EV with respect to all plausible ways of precisely evaluating coarse outcomes.

Where exactly does the UEV for a given strategy come from? As we might remember, the kinds of possibilities we conceive of are coarse-grained descriptions of many possible worlds, called hypotheses. So let’s construct a rough model that reflects the imprecision of our understanding of these hypotheses, taking inspiration from imprecise probabilities (i.e., representing beliefs with a set of probability distributions).

This will get a bit technical, but here’s the TL;DR:

Instead of pinning down a unique list of values for every hypothesis, we consider multiple ways of assigning precise values consistent with our evidence and principles. (E.g., to evaluate the hypothesis “misaligned ASI takes over”, we could entertain a range of more specific — though still coarse-grained — misalignment scenarios.)

Aren’t these precise values, too, arbitrary? Indeed. This model is meant to be merely the least bad formalization of our vague epistemic state (as discussed in the imprecision FAQ).

To compute a strategy’s UEV, we compute the EV wit

...
Invitation to an IRL retreat on AI x-risks & post-rationality in Ooty, India
by bhishma — 2025-06-08T13:21:42.512Z

Note: This post is an invite for the retreat, as well as an expression of interest for similar events which will be conducted. We are using the form for both.

TL;DR: Ooty AI Retreat 2.0 (June 23-30, '25, open to all): We're moving beyond e/acc vs doomer debates to practically test AI tools & opportunities. We'll cultivate post-rational flexibility (multiple perspectives, intuition, meditation) through coding, writing sprints, strategy talks, and more. Interested? Fill the form! (deadline : June 17th)

Hey folks,

We're running our Ooty AI Alignment Retreat 2.0 from June 23-30, 2025. The last one happened in June 2024. This is open to non technical people also!

When we say x risks, we don't only mean just threat models and p doom/takeoff discussions. We want to work on identifying and navigating the opportunities that current AI capabilities unlock, striving for nuanced understanding over simplistic e/acc versus doomer dichotomies. We welcome people from all kinds of backgrounds, worldviews, and hope to have a productive discussion. Crucially, we anchor these explorations with quick empirical tests and tight feedback loops with reality and today's models—engaging in a 'near mode' rather than getting lost in purely abstract futures.

When we say post rationality, we are cultivating the ability to hold multiple, sometimes even contradictory, mental models or "systems" at once, and fluidly switching between them depending on the situation. This often involves re-engaging with things previously dismissed as vestigial like intuition, emotions, embodiment, or even spiritual concepts—not as blind belief, but as potentially useful sources of information or ways to operate.

As the memetic environment becomes increasingly adversarial, having tools like metacognitive skills and contemplative technique are important to ensure coordination (alignment) with ourselves across time, integrating the different mind parts, giving us stability to handle these turbulent times better.

To get a sense of what happened last year,

Last year's events

Some of the sessions that we had were,

The future of Brain Machine Interfaces, discussion on bandwidth constraints, etc
Talks on Centaurism, Cyborgism and integration between AI and humans
Sahil gave an introduction to his agenda of Live Theory
Concrete predictions on the future of AI via manifold, forecasting

...
Litanies Of The Way
by Matthew McRedmond — 2025-06-08T07:32:14.264Z

A fun index of the concepts from The Sequences in the form of several litanies. These litanies are neither a guide nor an introduction to The Way of Rationality because rationality is an art not a series of precipes. Rather, these litanies serve as a tool; for meditating on, for connecting ideas; or for preparing for The Ritual Of Changing One’s Mind. For some this may be a useful exercise, for others not. If they help you, Enjoy!

Litanies Of The Way

Litany On Litanies

Litany Against Bias

Litany For Communication

Litany Against Politics

Litany For Inquiry

Litany For Letting Go

Litany For Belief

Litany Against The Dark

Litany For Seeing

Litany For Curiosity

Litany For Minds

Litany On Litanies

I will use these litanies to grow stronger

And when they are no longer useful I will not mourn

For although an apprentice must first imitate The Way that is known

A master must discover The Way that is to come

Litany Against Bias

When I learn about a bias
I will not use it as a Fully General Counterargument

And when I teach others of bias

I will first do no harm

When new evidence comes in

I will feel the burden of every detail

When I’m considering my plans

I will view them from the outside

When I want to be an effective altruist

I will not purchase satisfaction

When the world is chaotic
I will act lawfully

When there exist justifications beyond those given

I will reject the genetic heuristic

When I am presented with a false dilemma

I will look for a Third Alternative

And when I wish to overcome bias
I will know it is a worthy goal

Litany For Communication

When I feel I am misunderstood
I will be aware that my intent might not be transparent

When I need to communicate something complex

I will lay out an inferential pathway

When making propositions
I will not use Fake Justifications

When I am attacking propositions
I will give my True Rejection

Litany Against Politics

Politics is the Mind-Killer

When observing a debate
I will not treat arguments like soldiers

When I am dealing with a non-binary question
I will relinquish my scales and take up a notebook

When the problem is not just black and white

I will distinguish shades of grey
When I see someone kicking a vending machine

I will not hypothesize that they are a mutant

Litany For Inquiry

When I ponder
I will let the meaning choose my words

When I’m forming a hypothesis

I will look into the dark

When I question
I will Hug the Query

When I attack my b...
Make Data Pipelines Debuggable by Storing All Source References
by Brendan Long — 2025-06-08T04:16:36.149Z
Link: https://www.brendanlong.com/make-data-pipelines-debuggable-by-storing-all-source-references.html

A few jobs ago, I worked at company that collected data from disparate sources, then processed and deduplicated it into spreadsheets for ingestion by the data science and customer support teams. Some common questions the engineering team got were:

Why is the data in some input CSV missing in the output?
Why is data in the output CSV not matching what we expect?

To debug these problems, the process was to try to reverse engineer where the data came from, then try to guess which path that data took through the monolithic data processor.

This is the story of how we stopped doing that, and started storing references to all source data for every piece of output data.

(This is reconstructed from memory so I no longer claim 100% accuracy)

Get Source Data Into Your Database

The good news is that before I even started at this company, it was well understood that we needed to keep our data somewhere durable, since we might need to reprocess it one day. This involved putting all of the data in an S3 bucket, organized by source and date.

In order to tie our outputs to our inputs, we needed to first get our inputs into the database. We previously had some intermediate formats, but we wanted to know exactly where our data was coming from. So we added a source table:

csv_source

id	filename
1	client-x-2024-07-01.csv
2	client-x-2024-08-01.csv

csv_data

id	csv_source_id	first_name	last_name	job
10	1	Brendan	Long	Basket Weaver
11	2	Brendan	Long	Senior Basket Weaver
12	2	Example	McExampleton	Basket Weaver

(Ok, so we didn't literally upload the CSV and only uploaded it after determining the standardized column names, sort-of like this)

Tie Your Output Data to the Source Data

Now that we had our source data, so the next step was to tie our output data to it, so we could start to answer questions like "Where did this Brendan Long guy come from?" and "Where did this Basket Weaver job come from?". Our output data frequently needed to deduplicate data from multiple sources, and the obvious choice here would be to link to the source of the winning data, but for debugging, we don't just want to know about the winning data, we want to know about all of it.

So we added all all of the sources for a piece of data. We did this with join tables, but the examples will show them as inline arrays to keep this readable.

jobs

id	csv_data_ids	name
20	{10,12}	Basket Weaver
21	{11}	Senior Basket Weaver

employees

id	csv_data_ids	first_name	last_name	job_id

...
Letting Kids Be Outside
by jefftk — 2025-06-08T01:30:19.413Z

When our kids were 7 and 5 they started walking home from school
alone. We wrote explaining they were ready and giving permission, the
school had a few reasonable questions, and that was it. Just kids
walking home from the local public school like they have in this
neighborhood for generations.

Online, however, it's common for people to write as if this sort of
thing is long gone. Zvi captures a
common view:

You want to tell your kids, go out and play, be home by dinner, like
your father and his father before him. But if you do, or even if you
tell your kids to walk the two blocks to school, eventually a
policeman will show up at your house and warn you not to do it again,
or worse. And yes, you'll be the right legally, but what are you going
to do, risk a long and expensive legal fight? So here we are, and
either you supervise your kids all the time or say hello to a lot of
screens.

His post also references ~eight news stories where a family had
trouble with authorities because they let their kid do things that
should be ordinary, like walking to a store at age nine.

It's not just Zvi: parents who would like kids to have more freedom
often focus on the risk, with the potential for police or Child
Protective Services to get involved. While it's important to
understand and mitigate the risks, amplifying the rare stories that go
poorly magnifies their chilling effect and undermines the overall
effort.

I showed the quote to our oldest, now 11 and comfortable on her
own: "I sincerely doubt that a police officer would get mad at me for
walking to school or to the corner store by myself."

She got to this level of comfort by spending a lot of time out in our walkable
kid-friendly neighborhood. Sometimes with us, and increasingly on
her own. For example it's raining today and she just came back to the
house to tell me that she was grabbing rain gear and then she was
going puddle jumping with two younger neighborhood kids. In a bit
I'll stop writing and take her younger sister (age 3) out to join in.

Some other examples of being out alone:

Heading to a school concert the 8yo was running late and the
10yo was getting impatient. I asked her: "you know the way, do you
want to go on ahead by yourself?" She walked the half mile without
issue, with her watch as
backup.

Both older kids will go to the corner store to spend

...
LessOnline Could Use Meeting Stones
by Brendan Long — 2025-06-08T01:01:47.624Z

Back in the mists of time, there was this game called World of Warcraft, where you and your four closest friends would team up to complete dungeons together. To handle the inevitable people with no friends, they added meeting stones near each dungeon. When you clicked a meeting stone, it added you to a queue, and once five people were in the queue, you would all be added to a group together.

I was thinking abut this at LessOnline last week, since I don't have any friends sometimes had trouble finding groups of people to talk to. One way to handle this was to walk up to an existing group and talk to them, but this was tricky since sometimes all of the existing groups were 4+ people already. My strategy ended up being to sit on an empty couch in the busiest areas and wait for people to come to me, but sometimes there were no empty couches in the busy areas.

This made me wish we had some (virtual) meeting stones:

On WriteHaven, you click "Looking For Group".
Three[1] additional people click "Looking For Group".
WriteHaven notifies me that a group has been formed and gives me a randomly-selected location[2] that I've been summoned to.

This could also potentially be a non-WriteHaven app, and I'm tempted to just write it myself, but it would only really be useful if everyone at a conference was aware of it.

What do you all think?

^

The ideal conversation size might be smaller than four, but groups of two can lead to being matched with one other person who you don't actually end up liking, and not having an non-awkward way to extract yourself.

^

The conference organizers can setup a list of locations to randomly select from, like "The fire table near the Diagonal Hall".
MRI tracers
by bhauth — 2025-06-07T23:03:33.524Z
Link: https://www.bhauth.com/blog/biology/mri%20tracers.html

MRI scans and PET scans are different methods for medical imaging, and they observe different things:

MRI scans can see which nuclei are present in regions. Sometimes bond types of particular elements can be distinguished, eg in 1H NMR and fMRI.

PET scans add a short-lived radioactive tracer (usually organic molecules with radioactive fluorine added to them) and observe where decays happen. This can track things like movement of glucose in the body. The half-life of fluorine-18 is ~110 minutes. PET scans are often combined with CT or MRI to correlate tracers with locations of organs.

Of course, MRI scans are generally preferably because they don't expose patients or staff to radiation, and don't require short-lived radioactive compounds. That being the case, who among us hasn't asked:

Could we use tracer compounds for MRI scans that let MRI do what PET is used for?

Historically, such tracers for MRI weren't available, but there are now some interesting options. (Yes, metallic contrast agents (eg gadolinium) have been used with MRI, but that doesn't do what we need here.)

hyperpolarized carbon-13

1.1% of carbon is 13C, which is stable and basically harmless. Hyperpolarized 13C has a fairly strong MRI signal and can be used as a MRI tracer. You can make tracers using that by:

Getting some 13C.

Making some relevant molecule, like pyruvate or acetate. Unlike PET tracers, this doesn't need to be done right before they're used.

Polarizing the 13C. Basically, you get it very cold, and put it in a strong magnet until it goes all wibbly.

While the compounds are stable, the polarization decays quickly. Pyruvate spin lattice relaxation time (T1) is ~50-70 seconds ex vivo and ~20-30 seconds in vivo. Sufficient signal for detection is present for ~5x the T1. In that time, the tracer needs to be warmed up, injected, and detected by a scan.

So, you need a strong magnet, and special equipment for quickly processing the tracer during a MRI scan. Few hospitals are currently set up to do this, but technologically it's easier than the MRI scan itself.

nitroxide radicals

An obvious idea for MRI tracers is to use a metal contrast agent (eg gadolinium) in some organic complex attached to an antibody. That didn't work very well. This 1985 paper notes:

For monoclonal antibodies to function as selective MR contrast agents, substantial advances in technology must occur.

Peo...
Second order taste
by Adam Zerner — 2025-06-07T20:26:09.379Z

Last year I got my friend a gift card to Beast and Cleaver for his wedding, a local butcher shop and restaurant.

I was pretty proud of the gift. He and his wife said the meal was amazing and I wouldn't expect them to say that if it weren't true. Getting something on the wedding registry or perhaps a gift card would have been safer, but I don't think it would have yielded as much joy.

I can't take too much credit though. As a Yelp Elite food critic with over a decade of experience churning out high quality restaurant reviews, I like to think I have pretty good taste in food. But this time it wasn't me. It was Kenji.

I was watching a YouTube video by Kenji Lopez-Alt, a food writer who actually does have good taste in food. In his video he mentions Beast and Cleaver and says[1] that it's a very good butcher shop. I've read, watched and listened to a ton of Kenji's content over the years and think very highly of him.

So really what's happening here is something I'll call second order taste. First order taste would be me eating at Beast and Cleaver and determining it to be good. Second order taste is me identifying Kenji as someone with good first order taste.

I think that having good second order taste is pretty important. And I'm talking about taste more generally, not just as it relates to food. The world is complex. We can't evaluate everything ourselves. We don't have the time or the skills. For better or for worse, we often have to figure out who the trustworthy people are and listen to them.

Over the years I've identified a handful of people in various domains who I respect and see as highly trustworthy. Paul Ingraham for pain. Tori Olds for therapy. Ben Felix for personal finance. Ben Taylor for basketball. Demand Curve for marketing. Peter Attia for health. I think all of these people are quite smart and really know their field.

And then there's websites like The Best Bike Lock. I bought a moderately expensive e-bike recently. I store it in my apartment complex's bike room but historically there's been a fair amount of theft in that room, including my own electric scooter. I was dumb and only used a cable lock to secure the scooter, but there are also two U-locks that have been cut through and left sitting there in the bike room as well, so the theft is clearly going beyond low hanging fruit like cable locks.

Anyway, I wanted to do some research into how I ...
Dimensionalizing Forecast Value
by Jordan Rubin — 2025-06-07T18:45:26.923Z

tl;dr

In quantitative finance, the value of forecasting is obvious. No decisions are made without forecasts, and every forecast is created for some decision.

On

Metaculus

1

, the signal is just as sharp, but the market for the signal is fuzzier. The gap isn’t quality: it’s that the value proposition is implicit.

This post is my dimensionalization of forecast value: an identification of the unique factors that determine the importance of answering a question. These factors are Clarity, Leverage, and Efficiency (CLE).

I use CLE to decide when an answer will be valuable, before I spend time asking or answering the question.

To find value, I focus on questions where:

I might change my mind (Clarity)
A lot is at stake (Leverage)
Effort compounds over time (Efficiency)

Decisions Create Questions

Forecasts are useful for making decisions under uncertainty. This is a specific context. It is not always the relevant context.

Much existing content on forecasting assumes a question worth asking, and focuses on methodology improvements. Predicting the future is an unsolved, important problem; it’s good that smart people are working on these improvements.

But suppose I can get a decent forecast answer to any question about the future. What questions should I actually try to answer?

I don’t need a forecast if:

I am not making a decision
I already know what choice I want to make
It doesn’t matter what I decide

Even in a decision-making context, forecasts are also less helpful when:

I can’t observe the outcome of my decision
I can’t trust the forecaster
I have no time to decide

Forecasts are more helpful when:

The event repeats
Mistakes are highly visible
Catastrophic outcomes are possible
The decision matters
I don’t know what to do

How do I take this mess of factors and figure out what questions are worth answering?

Dimensionalizing Forecast Value: CLE

This section is a proposed framework for prioritizing forecasts.

There are many sources of uncertainty, and many decisions to make. How do I know what questions I want answered?

I will focus on three sources of value from forecasts:

Clarity: Forecasting can help me make a decision when I don’t know what to do.
Leverage: Forecasting can help me make the best decision when it matters most.
Efficiency: Forecasting now can help me make even better decisions next time.

These are the core forecasting value drivers

2

. They are intended to be roughly orthogonal, a...