trentmkelly/LessWrong-43k huggingface dataset:
Childhood and Education #10: Behaviors
by Zvi — 2025-06-23T20:40:07.158Z

Edition #9, that School is Hell, turned out to hit quite the nerve.

Thus, I’m going to continue with the system of making the roundups have more focus in their themes, with this one being the opposite of school questions, except for the question of banning phones in schools which seemed to fit.

Table of Contents

Metal Health.

Coercion.

Game Theoretically Sound Discipline.

The Joy of Doing Nothing.

ADHD Exists But So Do Boys.

Sports Go Sports.

On the Big Screen.

Kids Media Is Often Anti-Capitalist Propaganda.

Culture.

Travel.

Phone a Friend.

The Case For Phones.

Ban Cell Phones in Schools.

A Sobering Thought.

Metal Health

Henry Shevlin: I asked a high school teacher friend about the biggest change in teens over the past decade. His answer was interesting. He said whereas the ‘default state’ of teenage psychology used to be boredom, now it was anxiety.

Makes me wonder if in some deep psychosocial way there’s a trade off between the two emotions; eg, maybe boredom is essential for background anxiolytic mental processes (cf exposure to pathogens and allergies, the hygiene hypothesis).

Paul Graham: Does he have a guess about why this change occurred?

Henry Shevlin: I’ve just asked him and will let you know his response! Based our on previous conversations, I suspect he’ll say smartphones + social media as a uniquely toxic combination for teen mental health.

Reply from my friend. Basically – phones, social media, cultural doomerism, and decline of long-form literacy.

Friend: That’s about right. But also I think there’s a lot of gloom that bombards them without the social media stuff. Climate, politics, life opportunities, etc etc. Loss of literacy may be something too, reading long and in depth about things brings a degree of control.

If you are bored, you can now go on your phone until you are anxious instead. You could also make other choices, but that seems to be the default.

Coercion

I’m not always with Žižek, but here I’m with Žižek.

Violeta: Žižek on authoritarian parenting:

Asking a child to visit his grandmother because it’s his duty – he has no choice but to fulfill it – is *more* respecting of his inner freedom, less authoritarian than telling him it’s his choice BUT “think about how much grandma loves you.”

This goes so much farther than children. You see it in so many other situations as well.

It’s one thing for someone to have to do [X]....
Compressed Computation is (probably) not Computation in Superposition
by Jai Bhagat — 2025-06-23T19:35:42.443Z

This research was completed during the Mentorship for Alignment Research Students (MARS 2.0) Supervised Program for Alignment Research (SPAR spring 2025) programs. The team was supervised by Stefan (Apollo Research). Jai and Sara were the primary contributors, Stefan contributed ideas, ran final experiments and helped writing the post. Giorgi contributed in the early phases of the project. All results can be replicated using this codebase.

Summary

We investigate the toy model of Compressed Computation (CC), introduced by Braun et al. (2025), which is a model that seemingly computes more non-linear functions (100 target ReLU functions) than it has ReLU neurons (50). Our results cast doubt on whether the mechanism behind this toy model is indeed computing more functions than it has neurons: We find that the model performance solely relies on noisy labels, and that its performance advantage compared to baselines diminishes with lower noise.

Specifically, we show that the Braun et al. (2025) setup can be split into two loss terms: the ReLU task and a noise term that mixes the different input features ("mixing matrix"). We isolate these terms, and show that the optimal total loss increases as we reduce the magnitude of the mixing matrix. This suggests that the loss advantage of the trained model does not originate from a clever algorithm to compute the ReLU functions in superposition (computation in superposition, CiS), but from taking advantage of the noise. Additionally, we find that the directions represented by the trained model mainly lie in the subspace of the positive eigenvalues of the mixing matrix, suggesting that this matrix determines the learned solution. Finally we present a non-trained model derived from the mixing matrix which improves upon previous baselines. This model exhibits a similar performance profile as the trained model, but does not match all its properties.

While we have not been able to fully reverse-engineer the CC model, this work reveals several key mechanisms behind the model. Our results suggest that CC is likely not a suitable toy model of CiS.

Introduction

Superposition in neural networks, introduced by Elhage et al. (2022), describes how neural networks can represent N sparse features in a D<N-dimensional embedding space. They also introduce Computation in Superposition (CiS) as computation performed entirely ...
Situational Awareness: A One-Year Retrospective
by Nathan Delisle — 2025-06-23T19:15:58.708Z

tl;dr: Many critiques of *Situational Awareness* have been purely qualitative; one year later we can finally check the numbers. I did my best to verify his claims using public data through June 2025, and found that his estimates mostly check out.
This is inherently noisy work - nothing herein is certain, and I would encourage red-teaming, particularly in the algo-efficiencies/unhobbling/hardware sections.

Many thanks to Kai Williams, Egg Syntax, and Aaron Scher for their critical feedback.

Abstract

Leopold Aschenbrenner’s 2024 essay Situational Awareness forecasts AI progress from 2024 to 2027 in two groups: "drivers" (raw compute, algorithmic efficiency, and post-training capability enhancements known as "un-hobbling") and "indicators" (largest training cluster size, global AI investment, chip production, AI revenue, and electricity consumption).[1] Drivers and the largest cluster size are expected to grow about half an order of magnitude (≈3.2×) annually, infrastructure indicators roughly doubling annually (2× per year), with AI revenue doubling every six months (≈4× per year).[2]

Using publicly available data as of June 2025, this audit finds that global AI investment, electricity consumption, and chip production follow Aschenbrenner’s forecasts. Compute, algorithmic efficiency, and unhobbling gains seem to follow Aschenbrenner’s forecasts as well, although with more uncertainty. xAI’s Grok 3 exceeds expectations by about one-third of an order of magnitude.[3] However, recent OpenAI and Anthropic models trail raw-compute trends by about one-third to one-half an order of magnitude,[4] and AI-related revenue growth is several months behind. Overall, Aschenbrenner’s predicted pace of roughly half an order-of-magnitude annual progress is supported by available evidence, though measurement uncertainty persists.

Graph. At-a-glance scoreboard.

Introduction

“It is strikingly plausible that by 2027, models will be able to do the work of an AI researcher/engineer. That doesn’t require believing in sci-fi; it just requires believing in straight lines on a graph.” — Leopold Aschenbrenner

Ambitious AI forecasts, such as the AI Futures Project’s modal scenario that reaches artificial super-intelligence by 2027,[5] have generated a lot of debate recently. These forecasts often implicitly rely on extrapolation of recent trends, an approach that critics cl...
Recognizing Optimality
by jimmy — 2025-06-23T17:55:07.889Z

Short feedback loops are great when the feedback is clear, but without a guiding framework it can be hard to know what to make of the feedback -- especially in the short term. If things get uncomfortable, is that a bad thing or is that a counter-steering? If I'm getting results quickly, am I Goodharting to rapid yet unstable results? If I'm not getting results quickly is it because the problem is hard or just that I'm ineffective? Am I even wise enough to know what a good outcome looks like in this situation?

A physicist can take the mass of your car, power capability of the engine, and the height differential between the start and finish, and give you a lower bound for how quickly you can race up a mountain. This is useful because if the lower bound is 99% of your actual time, then you know that you have little to gain in "learning how to drive" and need to focus on improving the power to weight ratio of your car if you want better results. If the lower bound is 1%, then don't bother upgrading the car because the answer is all in learning how to drive.

Even without the inputs to the equation the physicist can also judge your efficiency by watching your driving, and give criteria to optimize. If the accelerator isn't pinned to the floor, first do that. If you're revving past the power peak, upshift. If you're under it, then downshift. Optimality, in this case, looks like keeping the engine as close as possible to the power peak at every moment when the tires aren't at their limit of traction. If your engine is rarely near its power peak, you're not going to approach this lower bound. If you're up against the constraints, you're doing all you can. What might this look like when applied to "psychological" problems? What are the constraints?

To explore this, let's look at the maximally simple example of learning how to shoot a handgun effectively.

When learning to shoot a handgun there's various knowledge and skills that required. Once you get past the basics of things like "how to hold the gun" and "how to align the sights", there are still skills to be honed in order to do it well. You have to practice by feel so that you can keep the gun aligned and find your sights quickly. You have to practice to get a feel for what kind of sight picture is acceptable. You have to practice to get the feel of how to smoothly press the trigger so as to not pull the gun off targe...
Comparing risk from internally-deployed AI to insider and outsider threats from humans
by Buck — 2025-06-23T17:47:21.408Z

I’ve been thinking a lot recently about the relationship between

AI control

and traditional computer security. Here’s one point that I think is important.

My understanding is that there's a big qualitative distinction between two ends of a spectrum of security work that organizations do, that I’ll call “security from outsiders” and “security from insiders”.

On the “security from outsiders” end of the spectrum, you have some security invariants you try to maintain entirely by restricting affordances with static, entirely automated systems. My sense is that this is most of how Facebook or AWS relates to its users: they want to ensure that, no matter what actions the users take on their user interfaces, they can't violate fundamental security properties. For example, no matter what text I enter into the "new post" field on Facebook, I shouldn't be able to access the private messages of an arbitrary user. And on AWS, no matter what code I run on my EC2 instance, I shouldn't be able to access someone else's private S3 bucket.

This type of security is very "static": invariants are enforced by code that doesn't involve human review and so don't rely on humans reacting properly to something. This is typically how you mitigate risks from your users--users typically only need to have certain restricted types of access to your systems, and they're very numerous, so you need to handle them with automated approaches. I sometimes call systems that aim to be safe no matter what inputs a user provides "manifestly safe".

In contrast, consider how Facebook and AWS relate to their engineers. These companies want to be robust to their engineers abusing their access to violate security properties; they could do this either by abusing administrator privileges (e.g. direct access to machines that have user data) or by adding malign behavior to code that they write. In order to mitigate these risks, companies use very different types of security techniques. They rely on multi-party authorization, where multiple engineers have to sign off on executing sensitive commands, or code review, where someone needs to review code before it's pushed to environments where it has access to sensitive data. These techniques fundamentally rely on human labor.

Insider threats are harder to handle in that your engineers need to have much more fine-grained access, but easier in that your engineers ar...
Foom & Doom 2: Technical alignment is hard
by Steven Byrnes — 2025-06-23T17:19:50.691Z

2.1 Summary & Table of contents

This is the second of a two-post series on foom (previous post) and doom (this post).

The last post talked about how I expect future AI to be different from present AI. This post will argue that, absent some future conceptual breakthrough, this future AI will be of a type that will be egregiously misaligned and scheming; a type that ruthlessly pursues goals with callous indifference to whether people, even its own programmers and users, live or die; and more generally a type of AI that is not even ‘slightly nice’.

I will particularly focus on exactly how and why I differ from the LLM-focused researchers who wind up with (from my perspective) bizarrely over-optimistic beliefs like “P(doom) ≲ 50%”.[1]

In particular, I will argue that these “optimists” are right that “Claude seems basically nice, by and large” is nonzero evidence for feeling good about current LLMs (with various caveats). But I think that future AIs will be disanalogous to current LLMs, and I will dive into exactly how and why, with a particular emphasis on how LLM pretraining is safer than reinforcement learning (RL).

(Note that I said “feeling good about current LLMs”, not “feeling good about current and future LLMs”! Many of the reasons for feeling good about current LLMs have been getting less and less applicable over time. More on this throughout the post, especially §2.3.5 and §2.9.1.)

Then as a bonus section at the end, I’ll turn around and argue the other side! While I think technical alignment is very much harder than the vast majority of LLM-focused researchers seem to think it is, I don’t think it’s quite as hard and intractable a problem as Eliezer Yudkowsky seems to think it is. So I’ll summarize a few of the cruxes that I think drive my disagreement with him.

Here’s the outline:

Section 2.2 briefly summarizes my expected future AI paradigm shift to “brain-like AGI”;
Then I proceed to three main reasons that I expect technical alignment to be very hard for future brain-like AGI:

Section 2.3 argues that the competent behavior we see from brains (and brain-like AGI) is determined entirely by reinforcement learning (RL), whereas the competent behavior we see in LLMs today is instead determined largely by imitative learning, which I re-dub “the magical transmutation of observations into behavior” to remind us that it is a strange algorithmi

...
Foom & Doom 1: “Brain in a box in a basement”
by Steven Byrnes — 2025-06-23T17:18:54.237Z

1.1 Series summary and Table of Contents

This is a two-post series on AI “foom” (this post) and “doom” (next post).

A decade or two ago, it was pretty common to discuss “foom & doom” scenarios, as advocated especially by Eliezer Yudkowsky. In a typical such scenario, a small team would build a system that would rocket (“foom”) from “unimpressive” to “Artificial Superintelligence” (ASI) within a very short time window (days, weeks, maybe months), involving very little compute (e.g. “brain in a box in a basement”), via recursive self-improvement. Absent some future technical breakthrough, the ASI would definitely be egregiously misaligned, without the slightest intrinsic interest in whether humans live or die. The ASI would be born into a world generally much like today’s, a world utterly unprepared for this new mega-mind. The extinction of humans (and every other species) would rapidly follow (“doom”). The ASI would then spend countless eons fulfilling its desires, desires which we humans would find to be bizarre and pointless.

Now, I don’t endorse every word of that foom & doom scenario above—for example, I don't think “foom” requires recursive self-improvement. But I’m in much closer agreement with that scenario than the vast majority of AI safety & alignment researchers today, who tend to see the “foom & doom” scenario above as somewhere between “extraordinarily unlikely” and “already falsified”!

Those researchers are not asking each other “is it true?”, but rather “lol, can you believe that some people used to believe that?”.[1] Oh well. Laugh all you want. It’s still what I believe.

Conversely, from my perspective as a foom & doomer, it’s the mainstream contemporary AI alignment discourse that feels increasingly foreign and strange. How, I ask myself, do so many seemingly reasonable people wind up with such wildly, bafflingly over-optimistic beliefs as “P(doom)≲50%”??

Anyway, my main goal in these two posts is to explore how I wind up in such a different place as most other alignment researchers do today, on the question of foom & doom. I don’t particularly expect to win skeptical readers over to my side, but would at least like to convey that foom & doom is a story that hangs together and deserves a modicum of consideration.

1.1.2 Should I stop reading if I expect LLMs to scale to ASI?

These posts are mainly ...
The Lies of Big Bug
by Bentham's Bulldog — 2025-06-23T16:03:58.603Z

Crosspost of this blog article.

The majority of farmed animals killed each year are insects, and this number is only expected to increase. By 2033, it’s

estimated

that around 5 trillion insects will be slaughtered annually—more than 50 times the number of cows, pigs, chickens, turkeys, and the like currently slaughtered. But insect farming is built on a dark secret: its foundational premises are all lies!

There has been a calculated plot by the insect farming industry to mislead the public. They know that if the public knew the truth about them, they’d never support subsidizing them. The insect farms can only thrive in darkness—shielded from public scrutiny.

Insect farming was promised as an environmentally-friendly alternative to meat. In reality, however, there’s virtually no consumer market for insects, so

the insect farming industry mostly feeds insects to farmed animals like chickens and fish

. Insect farming is not a competitor of traditional, environmentally-devastating factory-farming—it’s a complementary industry. For this reason, the insect farming industry is on track to be

bad for the environment

and to increase carbon emissions. A report by the

UK government

estimates that insect feed fed to animals has 13.5 times the carbon emissions of soy-based alternatives.

Further studies

have only confirmed this impression. Part of why the industry is so disastrous is that it must use lots of resources and energy feeding insects and keeping them at adequate temperatures to grow.

How does the insect industry challenge such allegations? Well, mostly they ignore them. In the rare cases they address them at all, they make mealy-mouthed (meal-wormy-mouthed???) statements about the importance of sustainability while ignoring the inconvenient facts about their environmental impact. Journalism articles talking about the promises of insect farming

uncritically quote

the CEO’s of insect companies—a bit like an article about the impact of oil on the environment uncritically quoting the head of an oil company.

They also cite their own

life-cycle assessments

. Basically, they pay people to analyze their environmental impact, and shockingly discover that they’re great for the environment. It’s no surprise that when an insect company pays someone to analyze their environmental impact, the people paid by the insect industry often find that insect farming is great for the e...
AI companies aren't planning to secure critical model weights
by Zach Stein-Perlman — 2025-06-23T16:00:01.726Z

AI companies' security plans, per their published safety policies, involve just SL3-4 security for powerful AI (see Appendix below).[1] (Moreover, companies are likely to not follow their current policies.)

Source.

Just SL3-4 is very inadequate; it would result in lots of extra x-risk relative to perfect security. If AIs that greatly accelerate AI research[2] can immediately be stolen, then the developer can't prioritize safety much without getting outraced (and the geopolitical situation is worse than if the US has a solid lead). It's much easier to tell a story where things go well if such model weights don't immediately proliferate.

Strong security isn't yet necessary. But you can't just decide to start operating with SL5 when you need it — it's likely prohibitively expensive to make the process faster than a couple years. Ideally companies would be working toward SL5 optionality — gaining the ability to quickly turn on SL5 when appropriate. Some kinds of security work will presumably be dramatically boosted by AIs before SL5 is necessary, so for now perhaps companies should focus on work that AI won't accelerate much and work that unlocks automatable security projects.

SL5 would be very costly, especially because it would slow development.[3] Unilaterally implementing it would make a company less competitive.[4] So companies aren't planning to do so; strong security requires coordination.[5]

To inform the world and potentially enable coordination, ideally companies would say what security would be appropriate if not for competition with other AI developers. And then when they deviate from that, they notice they're doing so, say so, and loudly say they wish for strong coordination where all frontier developers improve security. (I expect they are currently deviating, since I expect they'd say SL5 by 10x AI and it seems they're not on track.)

Appendix

OpenAI

OpenAI's Preparedness Framework defines a "High" security standard and says it will define a "Critical" security standard in the future. The High standard mentions reasonable security themes but it's sufficiently vague that it would be hard to tell if a company is meeting the standard and it's unclear what real-world security level the standard entails. OpenAI doesn't say which kinds of attacks or actors the standard is meant to protect against vs which are out of s...