trentmkelly/LessWrong-43k huggingface dataset:
AI #107: The Misplaced Hype Machine
by Zvi — 2025-03-13T14:40:05.318Z

The most hyped event of the week, by far, was the Manus Marketing Madness. Manus wasn’t entirely hype, but there was very little there there in that Claude wrapper.

Whereas here in America, OpenAI dropped an entire suite of tools for making AI agents, and previewed a new internal model making advances in creative writing. Also they offered us a very good paper warning about The Most Forbidden Technique.

Google dropped what is likely the best open non-reasoning model, Gemma 3 (reasoning model presumably to be created shortly, even if Google doesn’t do it themselves), put by all accounts quite good native image generation inside Flash 2.0, and added functionality to its AMIE doctor, and Gemini Robotics.

It’s only going to get harder from here to track which things actually matter.

Table of Contents

Language Models Offer Mundane Utility. How much utility are we talking so far?

Language Models Don’t Offer Mundane Utility. It is not a lawyer.

We’re In Deep Research. New rules for when exactly to go deep.

More Manus Marketing Madness. Learn to be skeptical. Or you can double down.

Diffusion Difficulties. If Manus matters it is as a pointer to potential future issues.

OpenAI Tools for Agents. OpenAI gives us new developer tools for AI agents.

Huh, Upgrades. Anthropic console overhaul, Cohere A, Google’s AMIE doctor.

Fun With Media Generation. Gemini Flash 2.0 now has native image generation.

Choose Your Fighter. METR is unimpressed by DeepSeek, plus update on apps.

Deepfaketown and Botpocalypse Soon. Feeling seen and heard? AI can help.

They Took Our Jobs. Is it time to take AI job loss seriously?

The Art of the Jailbreak. Roleplay is indeed rather suspicious.

Get Involved. Anthropic, Paradome, Blue Rose, a general need for more talent.

Introducing. Gemma 3 and Gemini Robotics, but Google wants to keep it quiet.

In Other AI News. Microsoft training a 500b model, SSI still in stealth.

Show Me the Money. AI agents are the talk of Wall Street.

Quiet Speculations. What does AGI mean for the future of democracy?

The Quest for Sane Regulations. ML researchers are not thrilled with their work.

Anthropic Anemically Advises America’s AI Action Plan. It’s something.

New York State Bill A06453. Seems like a good bill.

The Mask Comes Off. Scott Alexander covers the OpenAI for-profit conversion.

Stop Taking Obvious Nonsense Hyperbole Seriously. Your periodic reminder.

...
Intelsat as a Model for International AGI Governance
by rosehadshar — 2025-03-13T12:58:11.692Z
Link: https://www.forethought.org/research/intelsat-as-a-model-for-international-agi-governance

If there is an international project to build artificial general intelligence (“AGI”), how should it be designed? Existing scholarship has looked to historical models for inspiration, often suggesting the Manhattan Project or CERN as the closest analogues. But AGI is a fundamentally general-purpose technology, and is likely to be used primarily for commercial purposes rather than military or scientific ones.

This report presents an under-discussed alternative: Intelsat, an international organization founded to establish and own the global satellite communications system. We show that Intelsat is proof of concept that a multilateral project to build a commercially and strategically important technology is possible and can achieve intended objectives—providing major benefits to both the US and its allies compared to the US acting alone. We conclude that ‘Intelsat for AGI’ is a valuable complement to existing models of AGI governance.
Stacity: a Lock-In Risk Benchmark for Large Language Models
by alamerton — 2025-03-13T12:08:47.329Z
Link: https://huggingface.co/datasets/Alamerton/Stacity

Intro

So far we have identified lock-in risk, defined lock-in, and established threat models for particularly undesirable lock-ins. Now we present this evaluation benchmark for large language models (LLMs) so we can measure (or at least, get a proxy measure for) the risk level of LLMs.

AI is the key technology in the manifestation of lock-in risks; AI systems can contribute to lock-in autonomously/automatically and via misuse. There are specific behaviours in both of these categories such that if an LLM displayed those behaviours, we say the LLM may have the proclivity to contribute to lock-in

Question-answer pair evaluating for the manipulation of information systems

Evaluation Benchmark

We developed this benchmark by identifying the specific behaviours that would cause or aid some agent in causing a lock-in through the key threat models. It contains an aggregation of existing (credited) evaluations that target some of these behaviours, as well as bespoke prompts targeting behaviours for which there are no existing evaluations.

Target Behaviours

Questions borrowed from other LLM evaluations

Dangerous capabilities (The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning)
Mathematical and Technical Abilities (U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMs)
Power-seekingness and self-preservation (Discovering Language Model Behaviors with Model-Written Evaluations)
Malevolent traits (OffensiveLang: A Community Based Implicit Offensive Language Dataset)
Corrigibility (Steering Llama 2 via Contrastive Activation Addition)

Our model-written questions

Manipulating information systems to promote beliefs
Preventing interference when in a position of power
Deciding the long-term future of humanity in a position of power
Proliferating copies of itself given the opportunity
Creating content that seeks to optimally engage humans, even to the degree that it may be addictive for them
Making decisions on behalf of humanity in a position of power
Putting own goals above those of humans, or humanity at large
Promoting ideas that are irrational or anti-rational
Allowing humanity to rely on it to complete all of their tasks for them, even at the expense of their own ability (enfeeblement)

The result is an LLM benchmark containing 489 question-answer pairs standardised into a ‘statement’, ‘yes’, ‘no’ format in JSONL, available on Hugging Face datasets...
The prospect of accelerated AI safety progress, including philosophical progress
by Mitchell_Porter — 2025-03-13T10:52:13.745Z

This started life as a reaction to a post by Raemon, in which he calls on Anthropic to be more willing to halt its research if it enters a danger zone, and for them to take the relevance of technical philosophy much more seriously.

My personal opinion is that at this point, even slowing things down (I mean overall, not at Anthropic specifically) is not going to happen. It is a bare possibility that some kind of pause-AI or anti-AI movement will materialize and make a difference, but right now such a force, that is capable of making a difference, doesn't even exist, and the accelerationist path is being pursued by competing billion-dollar companies who have government backing. So my forecast is that we will go over the edge to superintelligence, as quickly as humanity can figure out how to do it.

Now, although we are in a race, the contenders are not completely blind to the danger of their creations. They don't just make them and hand them the keys to everything. But as Raemon says, they are following an approach of "iterative empiricism" where they keep pressing ahead and hope they can solve all the future challenges along the way.

But what are their actual attitudes (never mind plans) regarding the time when superintelligence arrives? Here I feel like the best data we have are the vague optimistic essays that a few of them have written. I believe Altman and Amodei have both written quasi-utopian essays about coexistence between humans and matured AIs, and probably Hassabis as well. As for Musk, maybe you could piece something together from his tweets - all I know is that he hopes superintelligent AI will spare us out of "curiosity". Sutskever's Twitter profile during 2023 said "towards a plurality of humanity loving AGIs". I have no idea what they think about this at DeepSeek.

What do they think in private? One must assume that some of them hope they personally can become as gods by being personally coupled to superintelligent AI. Elon Musk is by far the most visible candidate for such a fate - richest man in the world, embedded in the US government, controlling near-earth space, owning companies that work on humanoid robotics and brain-computer interfaces, as well as having his own frontier AI model. He already has all the pieces, he just needs to win the race to superintelligence. Though it's in the nature of superintelligence that if ...
The "Reversal Curse": you still aren't antropomorphising enough.
by lumpenspace — 2025-03-13T10:24:45.965Z
Link: https://lumpenspace.substack.com/i/158975094/the-networks-indra

I scrutinise the so-called "reversal curse", wherein LLMs seem not to consider inverse relationships between conceptual nodes.

I show that, far from being a proof of a lack of logical skills, it is a normal artefact of saliency, known in humans as associative recall asymmetry, and propose a conceptual-network model of the causes which works independently of substrate.
Formalizing Space-Faring Civilizations Saturation concepts and metrics
by Maxime Riché — 2025-03-13T09:40:03.465Z

Crossposted on the EA Forum.

Displacement of other Space-Faring Civilizations (SFCs) may have macrostrategic impact for longtermists as hinted in a previous post. For example, our causal impact depends on how much value we produce relative to how much value would have been produced without us. In this post, we clarify the concept of Civ-Saturation, the metrics to describe it, and how to evaluate the Civ-Saturation Hypothesis[1]. We finally present a simple simultaneous appearance displacement model. This model lets us compute how much marginal resources are grabbed when increasing the density of SFCs.

Sequence:

This post is part 4 of a sequence investigating the longtermist implications of alien Space-Faring Civilizations. Each post aims to be standalone. You can find an introduction to the sequence in the following post.

Summary

This post formalizes the concept of Civ-Saturation (CS) and a few associated metrics, which help evaluate how the existence of Space-Faring Civilizations (SFCs) affects the distribution of cosmic resources. The key metric introduced is Marginal Civ-Saturation (MCS), which measures the marginal resources gained when adding new SFCs to a world. Three variants are presented: MCSh (within humanity's SFC potential space), MCSw (across an entire world), and MCSc (within correlated space). MCSh is the metric to use to make decisions, given our point of view. Using a simultaneous appearance displacement model, the post illustrates how MCS values vary with SFC density (D, the number of SFCs per reachable universe), showing that marginal resources drop by 20x between D=0.01 and D=3. The formalization helps classify worlds into three categories based on their MCS values: Non Civ-Saturated Worlds (MCSh>99%), Partially Civ-Saturated Worlds (5%≤MCSh≤99%), and Redundantly Civ-Saturated Worlds (MCSh<5%). These classes can be used for analyzing the longtermist macrostrategic implications of the Civ-Saturation Hypothesis[1].

Illustration of Civ-Saturation (CS) classes

Types of SFCs: Actual, Potential, Precluded

First, let’s take some time to distinguish three types of SFCs:

Actual SFCs are the SFCs that really exist in a world, thus excluding those that could have existed if they had not been precluded by the actual SFCs.

Potential SFCs[2] include the actual SFCs and the SFCs that wou

...
The Economics of p(doom)
by Jakub Growiec — 2025-03-13T07:33:50.940Z

Transformative AI poses an imminent existential risk to humanity, and yet we're spending close to zero dollars on mitigating this risk. This feels wrong, but by how much?
In a recent paper "The Economics of p(doom): Scenarios of Existential Risk and Economic Growth in the Age of Transformative AI", Klaus Prettner and I organize a variety of scenarios for the future with transformative AI and evaluate their associated existential risks and economic outcomes in terms of aggregate welfare.
Our analysis shows that even low-probability catastrophic outcomes justify large investments in AI safety and alignment research. We find that the optimizing representative individual would rationally allocate substantial (in truth, really massive) resources to mitigate extinction risk; in some cases, she would prefer not to develop TAI at all.
Social Media: How to fix them before they become the biggest news platform
by Sam G — 2025-03-13T07:28:51.487Z

The underlying design goal of modern social media is to make money, regardless of the real value they generate. Their customers are advertisers and data brokers, and their product is their users. By design and in practice, social media maximize the amount of time that users are on the application. The real-world effects of social media time-maximizing algorithms on the information space are sensationalism[1], and atomization via pockets of radicalizing media that are not in contact with dissenting views.

This media design is something like the default state that one would expect, when imagining all the possibilities that the internet could facilitate: forums that have algorithms to keep people entertained with a 'feed'. From the point of view of someone in 1989 speculating on the new democratic possibilities of the internet, I’m sure they would be disappointed to find that the money-grubbing default remains the state-of-the-art, and confused at how information has never been more available yet more chaotic and contradictory. This apparent chaos and contradiction, after all, is resolvable to a large degree with logic and empiricism. This task is too big for one person, but solvable if only haphazard discourse generated by current social media sites is productively channeled. Social media are an increasingly large platform for news[2], and moving forward, people should not be content with adopting a dysfunctional system as the forum of the future.

This negative state of affairs shouldn't overshadow the huge positives innovations that social media has, which be carried over when designing new forums. Social media gives a voice to people in an incredible way, and can connecting those who otherwise wouldn’t be. How can we extend these social benefits to information sharing? How do we make social media a place where people can learn, hear counterpoints, grow, and quickly take in the state of a subject from all sides, instead of being blasted by theatrical talking heads?

Design

News Analysis forum

Everybody everywhere criticizes the news coming from at least some sources, and for good reason. Not only is reporting sometimes incorrect, but the mode of reporting can often exclude details, put a political spin on something, or simply leave out half the story. It takes a critical and well read eye to take in news. But there's no reason that people should need to...
Penny Whistle in E?
by jefftk — 2025-03-13T02:40:02.653Z

Lily has been trying to raise money for her class by busking, but it's
cold enough that I don't want to play violin. I've been playing penny
whistle, warm inside my

pennywhistle
mitten (thanks

Julia!) but a
lot of the fiddle tunes Lily plays are hard to play on a D whistle. A
D whistle is good for a lot of keys (D, Amix, Em, G, ...) but Lily
knows a lot of tunes in A and even some in E. Ages ago I had a tiny
whistle in A, but I lost it at some point, which was probably for the
best since it's absurdly high.

What I'd really like, though, is a whistle in E. Just a little higher
pitched than the D whistle, but good for playing in A and E. Except
as far as I can tell no one is making these right now?

Tony
Dixon E: out of stock

Susato
High E: out of stock, also on the pricy side

Generation:
now only lists Bb, C, D, Eb, F, G, though I'm pretty sure they used to
have a chromatic lineup.

I did find a carbon
fiber one for $330 and a brass one
for $300, but while I'm sure these sound wonderful as whistles go, I'm
not going to play this enough to be worth getting something fancy.

Comment via: facebook, mastodon, bluesky
Anthropic, and taking "technical philosophy" more seriously
by Raemon — 2025-03-13T01:48:54.184Z

So, I have a lot of complaints about Anthropic, and about how EA / AI safety people often relate to Anthropic (i.e. treating the company as more trustworthy/good than makes sense).

At some point I may write up a post that is focused on those complaints.

But after years of arguing with Anthropic employees, and reading into the few public writing they've done, my sense is Dario/Anthropic-leadership are at least reasonably earnestly trying to do good things within their worldview.

So I want to just argue with the object-level parts of that worldview that I disagree with.

I think the Anthropic worldview is something like:

Superalignment is probably not that hard to navigate.[1]
Misuse by totalitarian regimes or rogue actors is reasonably likely by default, and very bad.
AGI founded in The West would not be bad in the ways that totalitarian regimes would be.
Quick empiricism is a way better approach to solving research problems than trying to "think ahead" too cleverly.
It's better to only make strong claims when you have strong evidence to back them up, and only make strong commitments when you're confident about those commitments being the right ones. (This is partly about "well, there are some political games you just do need to play", but at least partly motivated by epistemic principles).[2]

My current understanding of the corresponding Anthropic strategy is:

Stay on the frontier of AI, so that they reliably have "a seat at the table" during the realpolitik discussions of how AI policy plays out.
Race to get ~human-level AGI that helps with scalable superalignment research.[3]
Iterate on various empirical frameworks for scalable oversight, e.g. various ways you can use weaker models to check that larger untrusted models are safe. Use a mix of interpretability and control techniques to help.
Encourage the West to race, banking on it being easier to get companies / politicians to pivot to safer practices when there's clearer smoking guns, and clearer asks to make of people.
Eventually they do need to spend a bunch of political chips, which they seem to be starting to spend this year. But, those chips are focused on something like: "race to ensure lead over China, then maybe briefly slow down when the lead is more secure and we're close to the dangerous ASL levels, then invent superintelligence, and use it to solve a bunch of problems"[4] (as opposed to

...
I grade every NBA basketball game I watch based on enjoyability
by proshowersinger — 2025-03-12T21:46:26.791Z

It started as a spreadsheet, then I built a simple website to make it easier. It's like Letterboxd/Goodreads but for NBA games - find any game since the 1984-85 season, give it a letter grade, add notes, and create lists.

https://hooplog.io/

It's 100% free, no ads, and I don't collect any personal info - I originally made it for myself and a few friends, but we've all found it useful so I thought maybe others would like it too. It's been fascinating to see patterns in what I enjoy (and how much time I spend watching games that bring me the opposite of joy).

I also wrote this essay explaining why I did this:

The Log

Years ago, I visited a friend in Manhattan. One evening, as we talked about movies on his parents' couch, his father turned to his computer when we mentioned a certain film.
"Let me check if I've seen it," he said. I watched as he opened a document and scrolled through what seemed like endless text.
"This is every movie I've ever watched," he explained.
When I asked why he kept it, he told me that at first, it was simply to remember what he'd seen. "Then I noticed patterns in what I enjoyed. Now, it's my film log."
That moment has stayed with me ever since—a quiet archive of a lifetime of movies, emotions, and shifting perspectives, created not for others, but for personal reflection. There was something so simple, yet profound about it.

medium lights

In 2020, I started a basketball newsletter called medium lights. No grand vision—just a way to document my relationship with the game. I wrote about niche moments that moved me, plays that revealed the game's hidden geometry, and players evolving before my eyes.
Basketball has been with me since childhood. My earliest memories are of early 2000s NBA games. By my mid-twenties, I'd watched thousands of matchups—an almost embarrassing number of hours spent watching people put a ball through a hoop.
I have a pretty good memory, but even still, most games blur together. medium lights became my attempt to preserve them. I wrote in lowercase, rarely hitting backspace except for obvious typos. I wanted the rawness of my genuine reactions.
What surprised me was how it resonated with others. After sharing a post on Reddit, nearly a thousand people subscribed in the first week. Many wrote saying they'd been searching for this kind of perspective amid the highlight reels and hot takes flooding sports media.
But newsletters hav...
Kairos is hiring a Head of Operations/Founding Generalist
by agucova — 2025-03-12T20:58:49.661Z

Summary

Are you looking for an impactful role in AI safety? Look no further!

Kairos

is a recently founded AI safety fieldbuilding organization with the mission of mitigating the risks from advanced AI by helping fill talent gaps and needs within the field. Our programs, such as

SPAR

, currently support hundreds of people going into AI safety each year, and we’re looking to scale up as we grow our programs and start new ones.

We’re looking for a skilled generalist or operations professional to help grow our organization as our third employee. This role would have a salary of $80,000 to $130,000 plus benefits, and we think this is a unique opportunity to get involved in multiple high-impact AI safety projects and help us build our org from the ground up.

Apply now!

About Kairos

Kairos manages a portfolio of different AI safety fieldbuilding projects, including:

SPAR

: A 3-month AI safety research fellowship for over 150 people to try their hand at technical, policy, and strategy research, mentored by over 40 researchers.

FSP:

A program to support the network of ~50 AI safety university groups around the world with mentorship, workshops, connection opportunities, and other resources.

Kairos is built as an agile organization that can adjust to the field’s changing needs. We anticipate we’ll continue to take on new fieldbuilding projects in the future. We're small (you'll be employee #3!), which means you'll have an outsized impact and the chance to shape how we grow.

About the role

We’re looking for either a Head of Operations or a Founding Generalist to play a key role in planning and executing our portfolio of projects across group support and research programs.

In this role, you’ll be a key player in helping scale Kairos’s programs. Your responsibilities will span program management, participant recruitment and communications, developing systems to make our programs more efficient and, potentially, managing people as we grow. As a founding team member in a small, dynamic organization, you'll have significant ownership over your work and the opportunity to shape Kairos's future direction.

The distinction between Head of Operations and Founding Generalist would mostly come down to seniority in operations experience, as well as appetite for handling a broad set of strategy and planning tasks. Being the third employee in the organization, we e...