trentmkelly/LessWrong-43k huggingface dataset:
My latest attempt to understand decision theory: I asked ChatGPT to debate me.
by bokov — 2025-01-13T19:37:48.481Z

Epistemic status: probably full of inaccuracies, which I hope to learn from if pointed out; hopefully gets the gist right in places

I haven't had much luck understanding what practical problems CDT has which need EDT, ADT, or UDT to solve. So I asked ChatGPT to spoon-feed it to me.

Here is the link to the actual chat:

https://chatgpt.com/share/67855cae-4294-8005-83ca-c899b5b11b69

Transcript, sorry about the clumsy manual formatting:

Initial prompt:

What are the practical, current real-life examples where evidential decision theory or acausal decision theory most outperform casual decision theory?

These must only rely on already existing technology and must not be redundant with each other. List the ones with the biggest difference in performance relative to CDT first. Make sure to only include examples that hold up assuming CDT accounting for the wholistic, long-term impact of a decision not just immediate risk/reward.

ChatGPT said:

Evidential Decision Theory (EDT) and Acausal Decision Theory (ADT) can sometimes outperform Causal Decision Theory (CDT), even when CDT is adjusted to account for holistic, long-term impacts. Below are practical, real-life examples where EDT or ADT might deliver better results, ranked by the magnitude of their relative performance advantage over CDT:

1. Coordination in Distributed Systems (Biggest Advantage)

Scenario: Distributed computer networks use algorithms to decide how to allocate resources, like bandwidth or computational power. EDT-like strategies (e.g., Tit-for-Tat or reciprocal altruism) outperform CDT because they optimize decisions based on expected responses of other systems, creating a self-reinforcing network of cooperation.

Why CDT Fails: CDT focuses solely on direct causation (e.g., what benefits this action provides) and may fail to anticipate or rely on the influence its decision has on the strategies of other nodes in the network.

Technology Involved: Blockchain consensus algorithms, P2P file-sharing systems.

2. Mutual Aid in Insurance Networks

Scenario: Mutual insurance networks, such as agricultural cooperatives or health cooperatives, benefit when members act to maximize the group’s collective resilience. EDT-like reasoning encourages each participant to insure against rare but catastrophic events because their behavior influences group norms, improving collective survival.
Why CDT Fails: CDT may ...
AI models inherently alter "human values." So, alignment-based AI safety approaches must better account for value drift
by bfitzgerald3132 — 2025-01-13T19:22:41.195Z

Hi all. This post outlines my concerns over the effectiveness of alignment-based solutions to AI safety problems. My basic worry is that any rearticulation of human values, whether by a human or an AI, necessarily reshapes them. If this is the case, alignment is not sufficient for AI safety, since an aligned AI still challenge humanity’s autonomy over its normative conceptions of ethics. Additionally, AI’s ability to represent institutions and its association with “science,” “technology,” and other nebulous (but revered) concepts gives a single model an unprecedented amount of influence over the totality of human values.

At the very least, value drift should be acknowledged as a structural issue with all alignment solutions. At most, solutions like computational governance should limit the provenance of advanced models as much as possible.

I put TL; DRs after each section - feel free to take advantage!

Intro and topic overview

There’s a lot of discussion in AI alignment literature about feedback loops, bias amplification, and the like. The idea is that if a model makes misaligned predictions, the model’s application reifies this bias, making it more significantly encoded in training data and, by extension, more prevalent in future predictions.

Brian Christian’s example is crime prediction models. Statistical modeling shows that “crime” (that is, crime as recorded through arrest records), is more likely in low income neighborhoods. But, police departments might react to these models by sending more police to poorer neighborhoods. Doing so heightens arrest rates, as it would in any neighborhood with a greater police presence, which heightens the disparity of recorded crimes between affluent and low-income neighborhoods. As a result, future iterations of crime prediction models correlate income level and arrest rates more significantly; police departments increase their presence in these neighborhoods; and the correlation of crime and income deepens further as the cycle continues.

This picture glosses over two key implications of feedback loops. One is that bias amplification impacts ideology. Take an example of bias amplification in Practical Deep Learning for Coders: social media content recommendation algorithms. YouTube, and other social media websites, learn a user’s preferences from their browsing history and offer new suggestions accordingly. But, ...
Chance is in the Map, not the Territory
by Daniel Herrmann — 2025-01-13T19:17:15.843Z

"There's a 70% chance of rain tomorrow," says the weather app on your phone. "There’s a 30% chance my flight will be delayed," posts a colleague on Slack. Scientific theories also include chances: “There’s a 50% chance of observing an electron with spin up,” or (less fundamental) “This is a fair die — the probability of it landing on 2 is one in six.”

We constantly talk about chances and probabilities, treating them as features of the world that we can discover and disagree about. And it seems you can be objectively wrong about the chances. The probability of a fair die landing on 2 REALLY is one in six, it seems, even if everybody in the world thought otherwise. But what exactly are these things called “chances”?

Readers on LessWrong are very familiar with the idea that many probabilities are best thought of as subjective degrees of belief. This idea comes from a few core people, including Bruno de Finetti. For de Finetti, probability was in the map, not the territory.

But perhaps this doesn’t capture how we talk about chance. For example, our degrees of belief need not equal the chances, if we are uncertain about the chances. But then what are these chances themselves? If we are uncertain about the bias of a coin, or the true underlying distribution in some environment, then we can use our uncertainty over those chances to generate our subjective probabilities over what we’ll observe.[1] But then we have these other probabilities — chances, distributions, propensities, etc. — to which we are assigning probabilities. What are these things?

Here we’ll show how we can keep everything useful about chance-based reasoning while dropping some problematic metaphysical assumptions. The key insight comes from work by, once again, de Finetti. De Finetti’s approach has been fleshed out in detail by Brian Skyrms. We’ll take a broadly Skyrmsian perspective here, in particular as given in his book Pragmatics and Empiricism. The core upshot is that we don't need to believe in chances as real things "out there" in the world to use chance effectively. Instead, we can understand chance through patterns and symmetries in our beliefs.

Two Ways to Deal with Chance

When philosophers and scientists have tried to make sense of chance, they've typically taken one of two approaches. The first tries to tell us what chance IS – maybe it's just long-run frequency, or maybe it's so...
Progress links and short notes, 2025-01-13
by jasoncrawford — 2025-01-13T18:35:21.426Z
Link: https://newsletter.rootsofprogress.org/p/links-and-short-notes-2025-01-13

Much of this content originated on social media. To follow news and announcements in a more timely fashion, follow me on Twitter, Threads, Bluesky, or Farcaster.

Contents

From me and RPI
Jobs and fellowships
Other opportunities
Events
Questions
Announcements
Commentary on the wildfires
Sam Altman: AI workers in 2025, superintelligence next
Never underestimate elasticity of supply
“The earnestness and diligence of smart technical people”
“Americans born on foreign soil”
Undaunted
Eli Dourado’s model of policy change
Stats
Links
AI
Inspiration
Politics
China biotech rising
Predictions about war
Why did we wait so long for the camera?
Housing without homebuilders
Charts
Fun

From me and RPI

2024 in review for me and RPI, in case you missed it, including my annual “highlights from things I read this year”
First batch of recorded talks from Progress Conference 2024 are available now. Special thanks to Freethink Media for their excellent work producing these

Jobs and fellowships

Epoch AI hiring a Technical Lead “to develop a next-generation computer-use benchmark at Epoch AI. This will be for evaluating real-world AI capabilities as FrontierMath is for mathematics” (@tamaybes)
“Funded year-long PhD student fellowship, combining non-partisan economic policy research & public service,” deadline Jan 30. Apply here (@heidilwilliams_)
“I'm hiring (part-time) a techno-optimist who is obsessed with curating ideas” (@julianweisser)

Other opportunities

Call for Focused Research Organization proposals in the UK. “Submit your concept paper by Feb 7 & full proposal by March 28.” (@Convergent_FROs). “Don't forget to scroll down … to the part where we have a ‘Request for FROs,’ with some ideas for inspiration” (@AdamMarblestone)
Stories We'd Like to Publish (Part II), from Asimov Press. “Last time we did this, we got ~200 pitches and commissioned just about everything on the list” (@NikoMcCarty)
“We should build an Our World in Data, but for biotechnology. … If this vision appeals to you, send me an email (niko@asimov.com), and I’ll help you get started” (@NikoMcCarty). Asimov Press might fund this project
“My #1 priority right now is finding spaces to launch Primers [microschools]. We have educators all over Florida, Alabama & Arizona eager to launch schools in August. If you have 3+ classrooms available in your church, community center, school, sy

...
Better antibodies by engineering targets, not engineering antibodies (Nabla Bio)
by Abhishaike Mahajan — 2025-01-13T15:05:35.261Z
Link: https://www.owlposting.com/p/better-antibodies-by-engineering

Note: Thank you to Surge Biswas (founder of Nabla Bio) for comments on this draft and and Dylan Reid (an investor into Nabla) for various antibody discussions! Also, thank you to Martin Pacesa for adding some insight on a paper of his I discuss here (his comments are included).

Introduction

Antibody design startups are singlehandedly the most common archetype of bio-ML startup out there. It’s understandable why — antibodies are derisked modalities, CDR loops driving antibody efficacy makes the whole structure more amenable to ML, and there’s a fair bit of pre-existing data there. But, because it’s also the most common form of company, it’s difficult to really differentiate one over the other.

If you squint, you could make out some vague distinguishing characteristics. Bighat Biosciences does a lot of multi-property optimization, Absci and Prescient have the strongest external research presence, and so on. But there is a vibe of uniformity. It’s nobody’s fault, that’s just the nature of any subfield that has a huge amount of money flowing into it; everyone quickly optimizes. And, unfortunately for those of us who enjoy some heterogeneity, most everyone arrives at the same local minima.

Because of that, I’ve never really wanted to write about any antibody company in particular. None of them felt like they had a sufficiently interesting story. All fine companies in their own right, but they all tell the same tale: great scientists, great high-throughput assays, great machine-learning, and so on.

This brings us to the topic of this essay: Nabla Bio.

As may be expected, from the earlier sections of this piece, they are an antibody design startup. Founded by Surge Biswas, an ex-Church Lab student who worked with ML-guided protein modeling during his PhD, and Frances Anastassacos, they were launched in 2021 and currently sit at 15 employees. Nabla, on the surface, looks materially indistinguishable from most other antibody design companies. And, in many ways, they are! If you visit their website, you’ll immediately see references to barcoding, parallel screening, and machine learning. It feels like much of the same story as everyone else.

It took me until a few months later for me to realize that Nabla has an interesting difference. Because Nabla is not only an antibody engineering company. They may say they are, and their partnerships may only include antibodies, but I d...
Zvi’s 2024 In Movies
by Zvi — 2025-01-13T13:40:05.488Z

Now that I am tracking all the movies I watch via Letterboxd, it seems worthwhile to go over the results at the end of the year, and look for lessons, patterns and highlights.

Table of Contents

The Rating Scale.

The Numbers.

Very Briefly on the Top Picks and Whether You Should See Them.

Movies Have Decreasing Marginal Returns in Practice.

Theaters are Awesome.

I Hate Spoilers With the Fire of a Thousand Suns.

Scott Sumner Picks Great American Movies Then Dislikes Them.

I Knew Before the Cards Were Even Turned Over.

Other Notes to Self to Remember.

Strong Opinions, Strongly Held: I Didn’t Like It.

Strong Opinions, Strongly Held: I Did Like It.

Megalopolis.

The Brutalist.

The Death of Award Shows.

On to 2025.

The Rating Scale

Letterboxd ratings go from 0.5-5. Here is how I interpret the rating scale.

You can find all my ratings and reviews on Letterboxd. I do revise from time to time. I encourage you to follow me there.

5: All-Time Great. I plan to happily rewatch this multiple times. If you are an adult and haven’t seen this, we need to fix that, potentially together, right away, no excuses.

4.5: Excellent. Would happily rewatch. Most people who watch movies frequently should see this movie without asking questions.

4: Great. Very glad I saw it. Would not mind a rewatch. If the concept here appeals to you, then you should definitely see it.

3.5: Very Good. Glad I saw it once. This added value to my life.

3: Good. It was fine, happy I saw it I guess, but missing it would also have been fine.

2.5: Okay. It was watchable, but actually watching it was a small mistake.

2: Bad. Disappointing. I immediately regret this decision. Kind of a waste.

1.5: Very Bad. If you caused this to exist, you should feel bad. But something’s here.

1: Atrocious. Total failure. Morbid curiosity is the only reason to finish this.

0.5: Crime Against Cinema. You didn’t even try to do the not-even-trying thing.

The key thresholds are: Happy I saw it equals 3+, and Rewatchable equals 4+.

Here is the overall slope of my ratings, across all films so far, which slants to the right because of rewatches, and is overall a standard bell curve after selection:

The Numbers

So here’s everything I watched in 2024 (plus the first week of 2025) that Letterboxd classified as released in 2024.

The rankings are in order, including within a tier.

The correlation of my rating...
Paper club: He et al. on modular arithmetic (part I)
by Dmitry Vaintrob — 2025-01-13T11:18:44.738Z

In this post we’ll be looking at the recent paper

“Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks”

by He et al. This post is partially a sequel to my earlier

post on grammars and subgrammars

, though it can be read independently. There will be a more technical part II.

I really like this paper. I tend to be pretty picky about papers, and find something to complain about in most of them (this will probably come up in future). I don’t have nitpicks about this paper. Every question that came up as I was reading and understanding this paper (other than questions that would require a significantly different or larger experiment, or a different slant of analysis) turned out to be answered in this paper. But more importantly, the paper does two things that characterize it as a great paper in my opinion:

It operationalizes and investigates (from the point of view of interpretability) the first interesting mechanistic model of a new behavior: namely a transition between “in-distribution” vs. “out-of-distribution” learning. This is a distinction that transposes to the “in-context learning” setting the difference between memorization and generalization.
It finds an excellent “minimal interesting” toy model for this phenomenon, where the behavior exhibits the full range of sophistication we expect but is (at least potentially) fully understandable from an interpretability point of view. More specifically, this model is analogous to (and again, in some sense an ‘in-context transposition of) the revolutionary interpretability paper

Nanda et al.

that does a detailed mechanistic interpretability analysis of “usual” memorization vs. generalization in neural nets[1].

The setup in He et al. is that of a small transformer trying to solve a particular mathematical (and deterministic) sequence completion problem consisting of residues mod a prime p (equal to 29 in the paper), and related to the modular addition task in

Nanda et al.

Each task depends on a secret “context parameter,” or latent, given by a pair of residues (a,b). The latents are never openly given, but can be deduced from context[2] via a modular arithmetic computation. The paper essentially studies how (and to what extent) the transformer learns to deduce the context from examples in a text.

Latent context parameters are a common phenomenon in...
Cast it into the fire! Destroy it!
by Aram Panasenco — 2025-01-13T07:30:19.356Z

We should only use AGI once to make it so that no one, including ourselves, can use it ever again.

I'm terrified of both getting atomized by nanobots and of my sense of morality disintegrating in Extremistan. We don't need AGI to create a post-scarcity society, cure cancer, solve climate change, build a Dyson sphere, colonize the galaxy, or any of the other sane things we're planning to use AGI for. It will take us hard work and time, but we can get there with the power of our own minds. In fact, we need that time to let our sense of morality adjust to our ever-changing reality. Even without AGI, most people already feel that technological progress is too fast for them to keep up.

Some of the greatest thinkers and writers of humanity have warned us of the danger and seductiveness of unlimited power. Take this passage from Tolkien and tell me it doesn't sound like most of the people you've heard talk about the wonderful things they're planning to do with AGI:

Already the Ring tempted him, gnawing at his will and reason. Wild fantasies arose in his mind; and he saw Samwise the Strong, Hero of the Age, striding with a flaming sword across the darkened land, and armies flocking to his call as he marched to the overthrow of Barad-dir. And then all the clouds rolled away, and the white sun shone, and at his command the vale of Gorgoroth became a garden of flowers and trees and brought forth fruit. He had only to put on the Ring and claim it for his own, and all this could be.

Lovecraft warned us of what would happen when our abilities outpaced our morality, when we ourselves would become powerful like cosmic horrors:

The time would be easy to know, for then mankind would have become as the Great Old Ones; free and wild and beyond good and evil, with laws and morals thrown aside and all men shouting and killing and revelling in joy. Then the liberated Old Ones would teach them new ways to shout and kill and revel and enjoy themselves, and all the earth would flame with a holocaust of ecstasy and freedom.

Humanity has nothing to gain from AGI, and everything to lose. We don't need AGI to have human values or to follow instructions in a friendly manner. We just need to figure out that one command to seal off that power forever - without disassembling ourselves in the process.

If Geoffrey Hinton, Elizier Yudkowsky, and other top AI researchers are wrong about the power and d...
Moderately More Than You Wanted To Know: Depressive Realism
by JustisMills — 2025-01-13T02:57:32.022Z
Link: https://justismills.substack.com/p/moderately-more-than-you-wanted-to

Depressive realism

is the idea that depressed people have more accurate beliefs than the general population. It’s a common factoid in “things I learned” lists, and often posited as a matter of settled science.

In this post, I’ll explore whether it’s true.

Where It Began

The depressive realism hypothesis was first studied by Lauren Alloy and Lyn Yvonne Abramson, in a paper called

Judgment of contingency in depressed and nondepressed students: Sadder but wiser?

. Undergraduates had access to a button, which might (or might not) influence whether a light came on. Depressed students - but not nondepressed students - were pretty good at guessing how much influence the button had over the light. Specifically, nondepressed students tended to think they had more control than they actually did.

In other words, the seminal study showed something a lot more specific than “depressed people are more accurate”. It showed that they were better at inferring their own degree of influence. Or to put a finer point on it, (mildly) depressed people suffer less from the illusion of control.

There are two lines of criticism I want to explore here, before moving on to other studies. First, is the study probably accurate? And second, if it is, does it imply what it seems to?

Methodology

To get the big one out of the way first: Alloy and Abramson’s sample only consisted of undergraduates. This is an extremely common approach for psychological studies, because the people doing the studies are professors and undergrads are close at hand. But undergraduates are not exactly representative of humanity writ large. Also, the entire field of psychology

is on fire

, so “business as usual” is a poor defense.[1]

Furthermore, the total number of undergraduates in the study was 288: 144 depressed, and 144 not depressed. Depression was diagnosed through the

Beck Depression Inventory

, which seems fine, but does mean we’re probably dealing with self-report, so mileage may vary there too.

Finally, the study was done in 1979. This is the same decade as the

Stanford Prison Experiment

, one of the most famous studies of all time and now on shaky enough ground that Vox calls it

straight up fraud

.

As far as seminal studies from the 1970s go, this one seems pretty good. There have been replications (we’ll get to those), and if there’d been fraud or misconduct it would be a pretty good scoop. But again, 288 undergraduat...