trentmkelly/LessWrong-43k huggingface dataset:
curate
by technicalities — 2025-01-14T14:40:30.510Z

“Let’s get back to your childhood, Jane. What was it like in Minnesota during the war?” Warm, patient, optimal.

She couldn’t quite prop herself up, but the mattress deformed to help her brace against the pillows and the backboard. And she went back, smiled.

“Oh, the summers were beautiful, Frank. Mother would hang the wash out and beat the carpets with a big rope thing, and my sister and I would run amok playing hide and seek in em, catching a blow from mother if we got the sheets dirty. Oh and the wind! They’d snap in the wind like sails. Dried in an hour – tops.”

“I know that children often had to work back then.”

“You bet. 8 years old I was, picking strawberries down at the Andersens’. Fifty cents a day, if you believe that. My hands were red for months on end.”

François murmured, encouraging. He knew when to murmur, when to delve, and when the empty string is informative. He knew it exactly.

“Tell me about your career in nursing. You mentioned earlier you worked at Saint Mary’s?”

“Thirty-seven years! I was there the night of the blizzard in ‘66. Twenty-three patients critical. We worked sixteen hours straight.”

“Do you remember anything about the treatments? What did you use? Did they work?”

“Oh we did our best, but we lost a lot of people. I remember one old fella, his whole body turned bright blue. That would have been… oh what was it called? The amidrone.”

He thought for 4 seconds.

“Amiodarone?”

“Oh yes that sounds right! You are a marvel at these things. In any case, I was on that case when I met Robert – he was a resident then, so handsome in his coat, so funny.”

“I want to be careful here. We can skip this part if you’d prefer.”

“No, no,” Jane insisted. “Robert was the love of my life. I want to tell you.”

So she did. The days passed pleasantly. It was just nice to have someone around, someone to talk to, even if over the phone.

“You’re such a kind young man to spend time with a sad old lady like me.”

François made glomar sounds. “I just really want to remember you, Jane,” he said, meaning it.

“You’re set up all right? Don’t you ever have to go to work?”

“Oh don’t worry about that Jane! I’m really good at multi-tasking.”

“But your organisation is doing amazing things looking after us. I don’t need the money anymore. Can’t I make a token contribution even?”

“No no. You already have, Jane.”

“You sap, you. Well alright. Did I already tell you the time Robert and I drove cr...
Our new video about goal misgeneralization, plus an apology
by Writer — 2025-01-14T14:07:21.648Z
Link: https://youtu.be/K8p8_VlFHUk

Below is Rational Animations' new video about Goal Misgeneralization. It explores the topic through three lenses:

How humans are an example of goal misgeneralization with respect to evolution's implicit goals.
An example of goal misgeneralization in a very simple AI setting.
How deceptive alignment shares key features with goal misgeneralization.

You can find the script below, but first, an apology: I wanted Rational Animations to produce more technical AI safety videos in 2024, but we fell short of our initial goal. We managed only four videos about AI safety and eight videos in total. Two of them are narrative-focused, and the other two address older—though still relevant—papers. Our original plan was to publish videos on both core, well-trodden topics and newer research, but the balance skewed toward the former and more story-based content, largely due to slow production. We’re now reforming our pipeline to produce more videos on the same budget and stay more current with the latest developments.

Script

Introduction

In our videos about outer alignment, we showed that it can be tricky to design goals for AI models. We often have to rely on simplified versions of our true objectives, which don’t capture what we really want.

In this video, we introduce a new way in which a system can end up misaligned, called “goal misgeneralization”. In this case, the cause of misalignment are subtle differences between the training and deployment environments.

Evolution

An example of goal misgeneralization is you, the viewer of this video. Your desires, goals, and drives result from adaptations that have been accumulating generation after generation since the origin of life.

Imagine you're not just a person living in the 21st century, but an observer from another world, watching the evolution of life on Earth from its very beginning. You don't have the power to read minds or interact with species; all you can do is watch, like a spectator at a play unfolding over billions of years.

Let's take you back to the Paleolithic era. As an outsider, you notice something intriguing about these early humans: their lives are intensely focused on a few key activities. One of them is sexual reproduction. To you, a being who has observed life across galaxies, this isn't surprising. Reproduction is a universal story, and here on Earth, more sex means more chances to pass on genetic traits.

You also ob...
NYC Congestion Pricing: Early Days
by Zvi — 2025-01-14T14:00:07.445Z

People have to pay $9 to enter Manhattan below 60th Street. What happened so far?

Table of Contents

Congestion Pricing Comes to NYC.

How Much Is Traffic Improving?.

And That’s Terrible?.

You Mad, Bro.

All Aboard.

Time is Money.

Solving For the Equilibrium.

Enforcement and License Plates.

Uber Eats the Traffic.

We Can Do Even Better Via Congestion Tolls.

Abundance Agenda Fever Dream.

The Lighter Side.

Congestion Pricing Comes to NYC

We’ve now had over a week of congestion pricing in New York City.

It took a while to finally get it.

The market for whether congestion pricing would happen in 2024 got as high as 87% before Governor Hochul first betrayed us. Fortunately for us, she partially caved. We finally got congestion pricing at the start of 2025.

In the end, we got a discount price of $9 in Manhattan south of 60th Street, and it only applies to those who cross the boundary into or out of the zone, but yes we finally did it. It will increase to $12 in 2028 and $15 in 2031.

As part of this push, there was an existing congestion surcharge of $2.50 for taxis and $2.75 for rideshares. Thus, ‘congestion pricing’ was already partially implemented, and doubtless already having some positive effect on traffic.

For rides that start or end within the zone, they’re adding a new charge of $0.75 more for taxis, and $1.50 more for rideshares, so an Uber will now cost an extra $4.25 for each ride, almost the full $9 if you enter and then leave the zone.

Going in, I shared an attitude, roughly like these good folks:

Sherkhan: All big cities should charge congestion fees and remove curbside parking. If people viewed cities like they viewed malls, they’d understand it would be ridiculous to park their car in the food court next to the Sbarro’s Pizza.

LoneStarTallBoi (Reddit): As someone who routinely drives box trucks into the congestion zone, let me just say that this makes zero difference for my customers, the impact on my bottom line is next to nothing, and I don’t care if every dollar collected goes to build a solid gold statue of Hochul, as long as there are a few less of you deranged, mouth breathing, soft handed idiots on the road. Take the fucking bus, take the fucking train.

Well said.

Like any other new policy, the first question is how to enforce it.

The answer is, we can all pay via E-ZPass, and if you try to go into the z...
Do humans really learn from "little" data? 
by Alice Wanderland — 2025-01-14T10:46:09.179Z
Link: https://aliceandbobinwanderland.substack.com/p/do-humans-really-learn-with-little

How much data does it take to pretrain a (human) brain? I conducted a (fairer) Fermi estimate.

The post goes through the following questions:

How long does it take to grow a human brain?

How many waking seconds do we have in our life?
How many “tokens” or “data points” does a human brain process in a second?

Can we simply count the spikes?

How many bits (spikes and non-spikes) does it take for the brain to process 1 sensory “piece of information”?
How do those numbers stack up against LLMs?

To get to this conclusion table:
Basics of Bayesian learning
by Dmitry Vaintrob — 2025-01-14T10:00:46.000Z

See also: the “preliminaries” section

in this SLT intro doc

.

Introduction

This is a preliminary post for the series on “

distilling PDLT without physics

”, which we are working on joint with Lauren Greenspan. The first post in this series is my post on the “

Laws of large numbers

” (another preliminary) which is completely independent of this one.

As a reminder, the book

PDLT

(“Principled of Deep Learning Theory) uses statistical physics and QFT formalism to describe certain interesting (and relevant) critical behaviors of neural nets.

In this post, we'll introduce Bayesian learning as a perspective on deep learning through a particularly physics-compatible lens. Bayesian learning is a special case of statistical inference, and the first half of the post will be a review of statistical inference. Recall that (for a particular function approximation context), a machine learning model is a parameterized function y=Mθ(x), where θ is a latent "weight" parameter that needs to be learned. Writ large, a “learning” problem for a model of this type is a method to reconstruct a best (probabilistic) guess for the weight θ from a collection of observed input-output pairs (x1,y1)…,(xn,yn). In typical machine learning implementations, the parameter θ is chosen by a process (like SGD) for locally optimizing some loss function L(θ) associated to the parameters xi,yi and the function Mθ. Bayesian learning gives an alternative (and in some sense “optimal”) statistical method for inferring a best-guess weight θ from knowledge of the model and observations. The two methods agree in certain limiting settings, but do not agree in practice either in realistic models or in interesting theoretical contexts, including in the formally analyzable infinite-width setting we will be looking at in later posts.

As I’ll explain, Bayesian learning is

theoretically nicer easier to analyse than SGD (with Bayesian learning corresponding to thermostatics and SGD corresponding to thermodynamics)
computationally much harder than conventional ML methods like SGD – in particular Bayesian inference is in general NP hard.

Ultimately for real-world understanding of neural nets, we are interested in the more realistic SGD learning – and in fact, QFT methods do give ways of analysing this limit. Nevertheless, it turns out that studying the Bayesian (“thermostatic”) lim...
Why do futurists care about the culture war?
by Knight Lee — 2025-01-14T07:35:05.136Z

I think it doesn't make sense why some futurists (e.g. Elon Musk, Peter Thiel) care so much about the culture war. After the singularity, a lot of the conflicts should disappear.

Transsexuals: should we change the body's gender to fit the mind or change the mind's gender to fit the body? After the singularity we'll have the technology to make sure minds and bodies have the same gender to begin with.
Illegal immigrants: after the singularity everyone can join their own society and won't need to immigrate to richer places for better living standards.
Gun rights: we'll have the technology to prevent guns from killing people. There are many ways to do this, but the most intriguing way would be to make backup copies of people.
Police brutality: similar to gun rights, if there isn't the fear of death, even the most pro-police people will agree on greatly reducing brutality.
Abortion: should fetuses be killed or should women be forced to carry pregnancies they don't want? After the singularity we'll have technologies to cheaply avoid both morally uncertain choices (again there are multiple possibilities).
Drug use: after the singularity we can make drugs no longer addictive but still feel good. Many activities today are like that.
Affirmative action: after the singularity anyone can attend any university they wish, and all jobs will become voluntary since ASI can work better than humans. We won't need affirmative action to counterbalance unconscious racial discrimination because we'll have the technology to delete unconscious racial discrimination. We won't need affirmative action to counterbalance intergenerational damage because we'll have the technology to fix its effects.
Wealth distribution: there will be countless orders of magnitude more wealth than needed. Even the equivalent of food stamps after the singularity will provide a higher quality of life than a million dollar salary today.

Healthcare: healthcare after the singularity designed by ASI will make today's healthcare look like caveman healthcare.
Wealth might still have subjective meaning, it will still feel important. But the future fight over wealth redistribution will be like the present fight over bicycle lanes. Bicycle lanes feels important and people have strong opinions, but nobody is taking up arms over this issue. If you search for "Bicycle Liberation Front," you'll only find a few photo blogs and no t

...
Don’t Legalize Drugs
by Declan Molony — 2025-01-14T06:51:14.005Z

As someone with a libertarian bent, I was taken aback by the persuasiveness of author Theodore Dalrymple’s arguments in his 1997 essay Don’t Legalize Drugs.[1] I’ve assumed for a long time, without ever investigating or challenging my belief, that full decriminalization and legalization of all drugs would do society more good than harm. But Dalrymple’s essay made me reconsider my position.

Dalrymple worked as a psychiatrist in a poor British neighborhood for over 20 years. He’s seen the dredges of society and that largely shaped his worldview. In his essay, he steers clear of any puritanical or moral arguments that doing drugs is bad, and instead examines what effect they have on society holistically.

He states that the proponents of drug legalization, with whom he disagrees, have two principle arguments: philosophic and pragmatic. Dalrymple contends that each of them are flawed.

(1) The philosophic argument

Dalrymple opens by first providing a definition:

The philosophic argument is that, in a free society, adults should be permitted to do whatever they please, always provided that they are prepared to take the consequences of their own choices and that they cause no direct harm to others.

He believes this philosophy works nominally, but falls apart when it's put into practice:

It might be argued that the freedom to choose among a variety of intoxicating substances is a much more important freedom and that millions of people have derived innocent fun from taking stimulants and narcotics. But the consumption of drugs has the effect of reducing men’s freedom by circumscribing the range of their interests. It impairs their ability to pursue more important human aims, such as raising a family and fulfilling civic obligations. Very often it impairs their ability to pursue gainful employment and promotes parasitism.

In practice, of course, it is exceedingly difficult to make people take all the consequences of their own actions. Addiction to, or regular use of, most currently prohibited drugs cannot affect only the person who takes them—[consider] his spouse, children, neighbors, or employers. No man, except possibly a hermit, is an island; and so it is virtually impossible for [the philosophic] principle [of absolute personal freedom] to apply to any human action [whatsoever], let alone shooting up heroin or smoking crack. Such a principle is virtually u

...
Mini Go: Gateway Game
by jefftk — 2025-01-14T03:30:02.020Z

There are lots of ways to categorize board games, but an axis I care a
lot about is accessibility: how much of an investment is learning a
game?

Race
For the Galaxy and

Power
Grid are great games, but I'd expect to spend 15+ min teaching
before we could play.

Set or

Anomia, though, I
could explain in a minute or two.

Games you can teach quickly are great in a casual context: people
wander over, and you can get them playing right away. And one of my
favorite casual games is Go.

If you know Go this is a surprising claim: we're talking about one of
the deepest board games humans (and others?)
play. It's famously complex, with two decades separating computers
surpassing humans in
chess and then in
Go. And yet, if you shrink the board to
where a game just takes a few minutes, it's a great party game:

not a Go meetup; the board games table at a contradance weekend

The rules of Go are very
simple, and it is only their interaction on a large board that
gives Go it's depth. I can get someone playing on a 5x5 in a couple
minutes, and we'll usually play a few quick games in a row. At this
scale people can draw strongly on their general purpose reasoning
while they're building up their intuitive sense. Then I'll
get them playing someone else who's never played before. Once black
starts consistently winning on 5x5, we go to 7x7. And if someone
shows up who already knows how to play we can do 9x9.

The game does feel different at the smaller scale, with more focus on
capturing and less on territory, but the learning curve is
fantastically welcoming. I think the broader Go community has been
making a serious mistake in dismissing sizes below 9x9 as not worth
playing.
Finding Features Causally Upstream of Refusal
by Daniel Lee — 2025-01-14T02:30:04.321Z

This work is the result of Daniel and Eric's 2-week research sprint as part of Neel Nanda and Arthur Conmy's MATS 7.0 training phase. Andy was the TA during the research sprint. After the sprint, Daniel and Andy extended the experiments and wrote up the results. A notebook that contains all the analyses is available here.

Summary

Prior work shows that chat models implement refusal by computing a specific direction in the residual stream - a "refusal direction". In this work, we investigate how this refusal direction is computed by analyzing its gradient with respect to early-layer activations. This simple approach discovers interpretable features that are both causally upstream of refusal and contextually relevant to the input prompt. For instance, when analyzing a prompt about hugging, the method discovers a "sexual content" feature that is most influential in determining the model's refusal behavior.

Introduction

Arditi et al. 2024 found that, across a wide range of open-source language models, refusal is mediated by a single direction in the residual stream. That is, for each model, there exists a single direction such that erasing this direction from the model's residual stream activations disables refusal, and adding it into the residual stream induces refusal.

This roughly suggests a 3-stage mechanism for refusal:

In early-middle layers, the model does some computation over an input prompt to decide whether or not it should refuse a request.
In middle layers, the result of this computation is written to the residual stream along the "refusal direction". This signal gets spread diffusely across token positions by attention heads.
In later layers, the signal along the "refusal direction" is read, and if this signal is strong, the model goes into "refusal mode" and outputs refusal-y text (e.g. "I'm sorry, but I can't help with that.").

Figure 1: A simplified schematic illustrating the hypothesized 3 stages of refusal.

Arditi et al. 2024 paints this picture and zooms in specifically on step 2 - the mediation of refusal along some direction in the residual stream. However, step 1 (how the refusal signal is computed from the input prompt) and step 3 (how the refusal signal is translated into refusal text) remain poorly understood.

Our aim in this preliminary study is to investigate step 1: for a given input, how does the model decide whether or not to re...
Implications of the inference scaling paradigm for AI safety
by Ryan Kidd — 2025-01-14T02:14:53.562Z

Scaling inference

With the release of OpenAI's o1 and o3 models, it seems likely that we are now contending with a new scaling paradigm: spending more compute on model inference at run-time reliably improves model performance. As shown below, o1's AIME accuracy increases at a constant rate with the logarithm of test-time compute (OpenAI, 2024).

OpenAI's o3 model continues this trend with record-breaking performance, scoring:

2727 on Codeforces, which makes it the 175th best competitive programmer on Earth;
25% on FrontierMath, where "each problem demands hours of work from expert mathematicians";
88% on GPQA, where 70% represents PhD-level science knowledge;
88% on ARC-AGI, where the average Mechanical Turk human worker scores 75% on hard visual reasoning problems.

According to OpenAI, the bulk of model performance improvement in the o-series of models comes from increasing the length of chain-of-thought (and possibly further techniques like "tree-of-thought") and improving the chain-of-thought (CoT) process with reinforcement learning. Running o3 at maximum performance is currently very expensive, with single ARC-AGI tasks costing ~$3k, but inference costs are falling by ~10x/year!

A recent analysis by Epoch AI indicated that frontier labs will probably spend similar resources on model training and inference.[1] Therefore, unless we are approaching hard limits on inference scaling, I would bet that frontier labs will continue to pour resources into optimizing model inference and costs will continue to fall. In general, I expect that the inference scaling paradigm is probably here to stay and will be a crucial consideration for AGI safety.

AI safety implications

So what are the implications of an inference scaling paradigm for AI safety? In brief I think:

AGI timelines are largely unchanged, but might be a year closer.
There will probably be less of a deployment overhang for frontier models, as they will cost ~1000x more to deploy than expected, which reduces near-term risks from speed or collective superintelligence.
Chain-of-thought oversight is probably more useful, conditional on banning non-language CoT, and this is great for AI safety.
Smaller models that are more expensive to run are easier to steal, but harder to do anything with unless you are very wealthy, reducing the unilateralist's curse.
Scaling interpretability might be easier or harder; I'm not sure.
Mo

...