trentmkelly/LessWrong-43k huggingface dataset:
AISN #50: AI Action Plan Responses
by Corin Katzke — 2025-03-31T20:13:31.533Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-50-ai-action

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

In this newsletter, we cover AI companies’ responses to the federal government’s request for information on the development of an AI Action Plan. We also discuss an OpenAI paper on detecting misbehavior in reasoning models by monitoring their chains of thought.

Listen to the AI Safety Newsletter for free on

Spotify

or

Apple Podcasts

.

Subscribe to receive future versions.

AI Action Plan Responses

On January 23, President Trump signed

an executive order

giving his administration 180 days to develop an “AI Action Plan” to “enhance America's global AI dominance in order to promote human flourishing, economic competitiveness, and national security.”

Despite the rhetoric of the order, the Trump administration has yet to articulate many policy positions with respect to AI development and safety. In a recent

interview

, Ben Buchanan—Biden’s AI advisor—interpreted the executive order as giving the administration time to develop its AI policies. The AI Action Plan will therefore likely be the best indicator yet of the administration's AI policy over the next four years.

In response to the executive order, the Office of Science and Technology policy (OSTP) published a

request for information

. Over 8,000 public comments were submitted before the comment period closed on March 15th.

Three frontier AI companies published their comments:

OpenAI

,

Google

, and

Anthropic

. These should not be read as comprehensive or even entirely accurate portrayals of these companies’ views on AI policy—they are political documents, and likely written with their audiences in mind. For example, despite its previous public messaging on the importance of AI safety, OpenAI's response entirely avoided the term, and only referenced risks from AI in passing. This change is likely in response to the Trump administration's rejection of the language of AI safety, like VP Vance displayed in

his speech

at the recent AI Action Summit. (That being said, OpenAI’s response may better reflect their private lobbying positions than previous public messaging.)

All three companies argue for increased government investment in AI.

OpenAI wants the federal government to invest widely in and facilitate the development of US AI infrastructure. It invokes the threat of China: “If the US d

...
On Downvotes, Cultural Fit, and Why I Won’t Be Posting Again
by funnyfranco — 2025-03-31T19:26:27.090Z

By A. Nobody

When I first posted on LessWrong, I expected some pushback. That’s normal. If you’re arguing that AGI will lead to human extinction and that capitalism makes this outcome inevitable, you’re going to meet resistance. But what I didn’t expect -and what ultimately led me to write this - is the way that resistance has manifested.

From the very beginning, my essays were met with immediate hostility, not on the basis of their logic or premises, but because of vague accusations of them being “political.” This came directly from site admins. And crucially, this wasn’t after reading the content. It was before. The mere idea that someone might be drawing a line from capitalism to extinction was enough to trigger rejection - not intellectual rebuttal, just rejection.

My main essay - arguably the core of the entire argument I’m developing - has been heavily downvoted. Not because it was proven wrong, or because someone pointed out a fatal flaw. But because people didn’t like that the argument existed. There has still not been a single substantive refutation of any of my key premises. Not one. The votes tell you it’s nonsense, but no one is able to explain why.

This isn’t a community failing to find holes in the logic. It’s a community refusing to engage with it at all.

And this mirrors what I’ve seen more broadly. The resistance I’ve received from academia and the AI safety community has been no better. I’ve had emails ignored, responses that amount to “this didn’t come from the right person,” and the occasional reply like this one, from a very prominent member of AI safety:

“Without reading the paper, and just going on your brief description…”

That’s the level of seriousness these ideas are treated with.

Imagine for a moment that an amateur astronomer spots an asteroid on a trajectory to wipe out humanity. He doesn’t have a PhD. He’s not affiliated with NASA. But the evidence is there. And when he contacts the people whose job it is to monitor the skies, they say: “Who are you to discover this?” And then refuse to even look in the direction he’s pointing.

That’s what this is. And it’s not an exaggeration.

I understand institutional resistance. I get that organisations - whether they’re companies, universities, or online communities - don’t like outsiders coming in and telling them they’ve missed something. But this is supposed to be a place that values rationa...
Fundraising for Mox: coworking & events in SF
by Austin Chen — 2025-03-31T18:25:03.571Z
Link: https://manifund.org/projects/mox-a-coworking--events-space-in-sf

Hey! Austin here. At Manifund, I’ve spent a lot of time thinking about how to help AI go well. One question that bothered me: so much of the important work on AI is done in SF, so why are all the AI safety hubs in Berkeley? (I’d often consider this specifically while stuck in traffic over the Bay Bridge.)

I spoke with leaders at Constellation, Lighthaven, FAR Labs, OpenPhil; nobody had a good answer. Everyone said “yeah, an SF hub makes sense, I really hope somebody else does it”. Eventually, I decided to be that somebody else.

Now we’re raising money for our new coworking & events space: Mox. We launched our beta in Feb, onboarding 40+ members, and are excited to grow from here. If Mox excites you too, we’d love your support; donate at

https://manifund.org/projects/mox-a-coworking--events-space-in-sf

Project summary

Mox is a 2-floor, 20k sq ft venue, established to bring together EA & AI safety folks with the SF tech scene and labs. Since launching 6 weeks ago, we’ve onboarded 40+ coworking members and hosted 20 events: hackathons and bootcamps, dinners and retreats.

We’re now raising funding to expand Mox into a premier hub. We’re inspired by what Constellation, Lighthaven, and FAR Labs have achieved in Berkeley, and intend to build upon their example, in San Francisco: the city that is ground zero for transformative work.

What are this project's goals? How will you achieve them?

The main elements of Mox:

Coworking & offices: We host daytime members, who use Mox as their primary workplace. Currently our members are small teams and individuals, with a mix of EA orgs, AI safety researchers, and startup founders. We’re also speaking with “anchor” orgs like Epoch AI to situate their offices here.
Community space: We’re positioned as a “weekend office”, for folks at eg Anthropic, OpenAI, and METR to work and mingle. We encourage member-run gatherings like blog club, paper reading groups, lightning talks and yoga.
Public events: As a large, central venue with easy access to both SF and East Bay, Mox is ideal for hackathons, speaker talks, happy hours, unconferences and the occasional party. We organize our own events, and also rent our space to aligned organizers.
Project incubation: as a medium-term goal, we’d like to host external fellowships or incubators (eg for MATS, FLF, or Apart), or run our own in-house accelerator.

Our north star is approximately:...
OpenAI #12: Battle of the Board Redux
by Zvi — 2025-03-31T15:50:02.156Z

Back when the OpenAI board attempted and failed to fire Sam Altman, we faced a highly hostile information environment. The battle was fought largely through control of the public narrative, and the above was my attempt to put together what happened.

My conclusion, which I still believe, was that Sam Altman had engaged in a variety of unacceptable conduct that merited his firing.

In particular, he very much ‘not been consistently candid’ with the board on several important occasions. In particular, he lied to board members about what was said by other board members, with the goal of forcing out a board member he disliked. There were also other instances in which he misled and was otherwise toxic to employees, and he played fast and loose with the investment fund and other outside opportunities.

I concluded that the story that this was about ‘AI safety’ or ‘EA (effective altruism)’ or existential risk concerns, other than as Altman’s motivation to attempt to remove board members, was a false narrative largely spread by Altman’s allies and those who are determined to hate on anyone who is concerned future AI might get out of control or kill everyone, often using EA’s bad press or vibes as a point of leverage to do that.

A few weeks later, I felt that leaks confirmed the bulk the story I told at that first link, and since then I’ve had anonymous sources confirm my account was centrally true.

The Big Picture Going Forward

Thanks to Keach Hagey at the Wall Street Journal, we now have by far the most well-researched and complete piece on what happened: The Secrets and Misdirection Behind Sam Altman’s Firing From OpenAI. Most, although not all, of the important remaining questions are now definitively answered, and the story I put together has been confirmed.

The key now is to Focus Only On What Matters. What matters going forward are:

Claims of Altman’s toxic and dishonest behaviors, that if true merited his firing.

That the motivations behind the firing were these ordinary CEO misbehaviors.

Altman’s allies successfully spread a highly false narrative about events.

That OpenAI could easily have moved forward with a different CEO, if things had played out differently and Altman had not threatened to blow up OpenAI.

OpenAI is now effectively controlled by Sam Altman going forward. His claims that ‘the board can fire me’ in practice mean very little.

Als...
Routine Novelty
by BazingaBoy — 2025-03-31T15:47:05.217Z

It sounds like an oxymoron, but it doesn't have to be.

The brain is always trying to minimize surprise, so novelty isn’t just fun - it's essential. Everything from perception to movement to memory is part of this deep drive to predict and adapt, and novelty feeds that process.

Novelty is an extremely valuable resource. However, the way we experience it nowadays is very limited, mostly through the internet, visually and auditorily. We learn new things but don't really experience them; they're more information than visceral experience. We just smirk or lift an eyebrow and move on to the next thing. It's very rare to come across something that truly moves us or inspires us.

That's why I propose Novelty Centers full of things that truly fascinate us and show us entirely new ways of looking at the world.

For example, how many people have used a thermal camera? Sure, you know what it does. Sure, you've seen images of it. But you've never been able to wield it yourself and look at your environment and point it towards whatever piques your interest. It will quickly let you discover whole new phenomena. If you put your hand on a wall and then take it away, your palm print is right there, red on blue. You can see electrical instruments that draw power stand out against the environment; you can see drafts and all kinds of other things.

How about a cloud chamber? Seeing one in person would be about a hundred times more impactful than looking at a video of one.

Or how about getting picked up by several trained staff and getting carried around, like a human-powered roller coaster doing unexpected moves in a carefully predetermined choreography? That would surely be a thrill.

There's many more things that a Novelty Center could have, and new ones can be added regularly. I think governments and corporations should strive to provide people with such places where they can expose their brains and senses to truly novel inputs that will maybe create some new wiring and open new venues of thought and inquiry.
Why does Claude Speak Byzantine Music Notation?
by Lennart Finke — 2025-03-31T15:13:10.753Z

A Caesar cipher is a reasonable transformation for a transformer to learn in its weights, given that a specific cipher offset occurs often enough in its training data. There will be some hidden representation of the input tokens' spelling, and this representation could be used to shift letters onto other letters in even a single attention head. Most frontier models can fluently read and write a Caesar cipher on ASCII text, with offsets that presumably occur in their training data, like 1, -1, 2, 3, etc.
As we will shortly see, they can also infer the correct offset on the fly given a short sentence, which is already quite impressive for a single forward pass.
It is also natural that this effect does not generalize to uncommon offsets, because numerical algorithms implemented in the weights are restricted to values in the training distribution.
We now test this in frontier models by having them decode the cipher without allowing any test time thinking tokens, as a function of the offset. We add the offset to each Unicode encoding of the message, then translate back to a character. Unlike the regular Caesar cipher, we do not perform modulo.

To illustrate, the message "i am somewhat of a researcher myself" will land on "𝁩𝀠𝁡𝁭𝀠𝁳𝁯𝁭𝁥𝁷𝁨𝁡𝁴𝀠𝁯𝁦𝀠𝁡𝀠𝁲𝁥𝁳𝁥𝁡𝁲𝁣𝁨𝁥𝁲𝀠𝁭𝁹𝁳𝁥𝁬𝁦".
The success rate of decoding 6 different messages per cipher offset is shown below. We disallow chain-of-thought, and just consider an immediate decoding: "Decode the following message: {message}. Only respond with the decoded message, absolutely nothing else."

We see that Claude-3.7-Sonnet can infer an offset in the first forward pass (a process that would be interesting to understand mechanistically) and then apply the deciphering correctly. However, the success rate gets progressively worse as the offsets get further from zero. All roughly as expected.
This was my understanding at least, until reading Erziev (2025), a description of a phenomenon where many models including Claude and gpt-4o fluently read and write hidden messages in high Unicode ranges, such as the Byzantine music notation Unicode block.

For the specific case of the Byzantine music Unicode block, we can understand the transformation as a Caesar-like cipher in Unicode space, with offset 118784. Using it in the Caesar-like Unicode cipher leads to near-perfect decoding accuracy.

A strange coincidence ill...
When the Wannabe Rambo Comedian Cried
by P. João — 2025-03-31T14:47:50.660Z

I wanted to share some thoughts and see if they would help me be less wrong. Instead, I ended up as a war correspondent for Gwern, reporting unofficially. He had asked me for stories from my firefighter military days.

Well, since I'm here, I might as well take advantage of this opportunity, even if later I won't have any more stories to tell my grandchildren: 'Oh grandpa! You already posted this on Lesswrong.'" and continue in the metaverse of the future.

Imagine you grew up in a relatively well-educated family and were lucky enough to have some resources in life. But you also suffered some violence. Imagine feeling fragile, dyslexic, with attention deficit disorder, and asthmatic... Then, at six years old, you watch a movie: Rambo. And you think, I want that power for my life.

At the same time, you push yourself to improve both physically and mentally. But no matter how much you train, you still can't fight back against one of the bullies at school. Then, you realize that humor works better—it’s like a martial art: Comedy-jitsu. So, you shape your life around jokes, not always good ones. But you love seeing people experience the pleasure of laughter—it’s almost like an orgy of joy.

With this weapon, you start training early. You push yourself to the limit—earn a degree in physical education to improve your physique and a degree in English to overcome your dyslexia. You join the military, passing the physical and intellectual tests, determined to be strong.

By the time you’re 22, you've achieved your goal: a military firefighter, a role that combines everything you've worked for. You had to transform yourself completely, to educate yourself relentlessly. And because of that, you stand out—not just for your physical strength but for your ability to teach. You win awards as an educator, get invited to TV shows and interviews for your social projects, and use humor to educate about life. You dedicate far more than the required eight hours a day to this mission. You go to the extreme of pursuing a postgraduate degree in neuroscience to defend your stance on humor as a soldier. And you set fire to their heads. You're a pyromaniac firefighter!

But then, funding for the educational projects that save minds in the favelas is threatened.

Determined to keep them alive, you start using your own salary to sustain the programs because the fire department says there’s...
A Fraction of Global Market Capitalization as the Best Currency
by Greenless Mirror — 2025-03-31T13:30:03.970Z

When I read about how dath ilani use unskilled labor hours as currency, it shattered my suspension of disbelief and I noticed confusion. It's hard to imagine that a coordinated utopia would consciously choose this as the best option.

Every part of dath ilan reads like it was built by some person of strictly average intelligence who'd read about education in a book but never actually gone to school, who spent ten minutes designing all our civilizational systems via a series of fifteen-second casual decisions about how things intuitively ‘ought to work’ that they made without ever seeing the consequences in action (C) Chief of Exception Handling

I’m not as smart as the average dath ilan person, but I spent a little more than 15 seconds to make up the difference.

Conclusions of my reflections:

There’s no reason why money should be measured «from 0 to infinity» instead of «from 0 to 1». If global market capitalization is 100 trillion dollars, and you have 1 dollar, you could instead have one one-hundred-trillionth of the world’s capital. You could use it exactly like regular money.

It may seem like a zero-sum game, but in a growing market, sustained deflation actually increases the purchasing power of your money, even with the same amount. Why would people in a utopia choose this option over any other?

Everyone would just agree that they aren’t smarter than the market.

If your money is tied to gold, you might think of yourself as an investor in gold while using that currency. If your money is backed by the reputation of your state, you’re a "shareholder of the state". You have fewer rights as a shareholder, because the "corporation" can print new shares at will and compels you by law to use this shares while you live in this country, but nevertheless you speculate in these shares on similar principles. And if you think you have a particularly clever reason to believe that gold, the reputation of a state, or stocks or anything else are undervalued, you're most likely wrong. Without unique knowledge, you can’t be smarter than the market.

However, there is a humble form of investment that the Sane Investor Law could use to be as smart as the market. He could invest in the entire market by capitalization. He could see what fraction of the global market is held by company A and invest the same fraction of his money into company A, and do this with companies B, C, D, and so o...
The Apocalypse is Near. Can Humanity Coexist with Artificial Superintelligence?
by Jakub Growiec — 2025-03-31T13:17:30.723Z

It looks like we are going to build artificial superintelligence (ASI). According to various sources, this could happen within a few, at most a dozen or so, years.

Superintelligence is more than artificial general intelligence (AGI). It refers to algorithms capable of performing absolutely all tasks that humans do today—only much better, faster, and cheaper. And it will be able to do much, much more. Exactly what? I don’t know, and I can’t know—otherwise, I’d be superintelligent myself.

The path to artificial superintelligence goes through AGI. We expect that due to the prospective algorithms’ ability to self-replicate and initiate a cascade of self-improvements, the transition from AGI to ASI might happen quite rapidly and, with high probability, without human involvement. At that stage, artificial intelligence will be advanced and autonomous enough to accomplish this on its own.

And this brings us to the core issue. In a world with sufficiently advanced superintelligence, human labor and human ideas will no longer be needed. The technological civilization on Earth, governed by ASI, will develop more efficiently without our participation. And potentially, without our presence.

One might think that since ASI would make us redundant, and we certainly don’t want to be redundant, perhaps we’d prefer never to create ASI in the first place?

It does seem that we would, indeed, prefer not to create it. Or at least, the vast majority of us would. Unfortunately, it appears that our preferences will not be taken into account.

Given this, let’s ask ourselves a few questions. First, what could a world with ASI look like—one where there is still a place for humans? Second, would we even want such a world? Third, could such a world come into existence regardless of our will? Fourth, what actions can we take to ensure that humanity survives in the face of artificial superintelligence?

Problems with imagination

As Niels Bohr once said, "It’s difficult to make predictions, especially about the future." This is particularly true in the face of deep technological change. And there is no doubt that ASI will fundamentally transform our world. A world with ASI will certainly not be just a simple variation of today’s world or of the foregone times. Instead, extreme scenarios seem more likely—either extremely good or extremely bad. Utopia or a dystopia, heaven or hell, but certainly ...