trentmkelly/LessWrong-43k huggingface dataset:
Distillation of Meta's Large Concept Models Paper
by NickyP — 2025-03-04T17:33:40.116Z

Note: I had this as a draft for a while. I think it is accurate, but there may be errors. I am not in any way affiliated with the authors of the paper.

Below I briefly discuss the "Large Concept Models" paper released by Meta, which tries to change some of the paradigm of doing language modelling. It has some limitations that are not present for normal language models, but I read spent the time to read the paper in relative depth so I am here to provide a brief summary of it.

"Large Concept Models" (LCM) Paper

Large Concept Models aim to be a way to "improve language modelling" by "being more hierarchical". I think the easiest way to explain is to compare to normal decoder-only language models.

Normal language model (LLM):

Take text
Split the text into "tokens"
Embed the tokens into vectors
Pass token embed vectors into a model
The model outputs a probability distribution over potential new tokens

Large concept models (LCM):

Take a text
Split the text into sentences or "concepts". (They add a maximum limit so sentences can be split up into concepts)
Use a semantic embedding model, in particular SONAR, to embed the sentences into "concept vectors"
Pass the concept embed vectors into a model
The model somehow outputs a new sentence embed vector.
the new sentence embed vector can be decoded using SONAR again.

A normal LLM works by passing in and getting out single tokens. For the LCM, we instead pass and get out semantic vectors. The key model that makes this possible is SONAR, which is a text auto-encoder.

The main benefit I can see is that it likely is much better at long-contexts. Otherwise, it comes with some disadvantages.

ARCHITECTURES

The key difficulty: How to output new sentence embed vectors, since it a continuous space. They try a few approaches.

BASE LCM: They try the base case of directly predicting the expected next semantic vector using MSE.

The Base LCM model does not work well.

Quant LCM: In this approach[1], they try to make the sampling space of embedding vectors discrete by doing "Residual Vector Quantization". That is, they try to make it such that the semantic embed vector can be written as a decomposition of 64 components/"notebooks", each of which can be one of 8192 vectors/"units". These notebooks are also ordered by importance.

That is, a vector x would be written in terms of "units" from notebooks nb_1, ..., nb_64 like so:
x = nb_1[idx_1] + nb_2[idx_2]

...
Top AI safety newsletters, books, podcasts, etc – new AISafety.com resource
by Bryce Robertson — 2025-03-04T17:01:18.758Z

Keeping up to date with rapid developments in AI/AI safety can be challenging. In addition, many AI safety newcomers want to learn more about the field through specific formats e.g. books or videos.

To address both of these needs, we’ve added a

Stay Informed

page to

AISafety.com

.

It lists our top recommended sources for learning more and staying up to date across a variety of formats:

Articles
Blogs
Books
Forums
Newsletters
Podcasts
Twitter/X accounts
YouTube channels

You can filter the sources by format, making it easy to find, for instance, a list of top blogs to follow. We think this page might be particularly useful as a convenient place for field-builders to direct people to when asked about top books/newsletters/blogs etc.

As with all resources on

AISafety.com

, we’re committed to making sure the data on this page is high quality and current. If you think there’s something that should be added (or removed) please let us know in the comments or via the

general feedback form

.

The site now has the following resources:

We’d love to hear any ideas for other resources we should consider adding.
2028 Should Not Be AI Safety's First Foray Into Politics
by Jesse Richardson — 2025-03-04T16:46:37.370Z

I liked the idea in this comment that it could be impactful to have someone run for President in 2028 on an AI notkilleveryoneism platform. Even better would be for them to run on a shared platform with numerous candidates for Congress, ideally from both parties. I don't think it's particularly likely to work, or even get off the ground, but it seems worthwhile to explore, given that we don't know what the state of play will be by then. In my view, either the 2024 or the 2028 US Presidential election is probably the most important election in human history, and it's too late to affect the former.

My suggestion is that, if you're someone who is at all on board with this idea, as of March 2025, it is not sufficient to wait until the Presidential race gets going sometime in 2027 to do something. In particular, the first major test of political support for AI safety should not be the one time it has to work (note that I have pretty short timelines so I'm implicitly assuming that transformative AI will arrive before 2033).

Think about all the relevant knowledge that a practitioner of regular left-right politics might have, that someone interested in AI notkilleveryoneism politics mostly wouldn't have today:

Who is our base?
Who are the swing voters?
Who are the most relevant groups & institutions to have on our side?
What level of support can we expect i.e. how high or low should we aim?
Which kinds of messages work, which don't?
On which issues do voters trust us?
Which of our people are the most well-liked, and why?
What are the reliable failure modes for our brand of politics?
How would our opponents answer the above questions, and how can we exploit our knowledge of their answers?

If you lack clear answers to these questions, you're not doomed to fail, but you're not exactly setting yourself up for success. I also think that these are questions that are unlikely to be satisfactorily answered by armchair theorizing. The best way to answer them is to actually run an electoral campaign, a public-facing lobbying effort, something of this nature. SB1047 was a good start, let's do more.

Maybe you think the idea of AI safety having any kind of impact on the 2028 Presidential election is a pipe dream, in which case this post is not for you. But if you do want to leave the door open to this possibility, the time to start laying groundwork is now. The 2026 midterm e...
How Much Are LLMs Actually Boosting Real-World Programmer Productivity?
by Thane Ruthenis — 2025-03-04T16:23:39.296Z

LLM-based coding-assistance tools have been out for ~2 years now. Many developers have been reporting that this is dramatically increasing their productivity, up to 5x'ing/10x'ing it.

It seems clear that this multiplier isn't field-wide, at least. There's no corresponding increase in output, after all.

This would make sense. If you're doing anything nontrivial (i. e., anything other than adding minor boilerplate features to your codebase), LLM tools are fiddly. Out-of-the-box solutions don't Just Work for that purpose. You need to significantly adjust your workflow to make use of them, if that's even possible. Most programmers wouldn't know how to do that/wouldn't care to bother.

It's therefore reasonable to assume that a 5x/10x greater output, if it exists, is unevenly distributed, mostly affecting power users/people particularly talented at using LLMs.

Empirically, we likewise don't seem to be living in the world where the whole software industry is suddenly 5-10 times more productive. It'll have been the case for 1-2 years now, and I, at least, have felt approximately zero impact. I don't see 5-10x more useful features in the software I use, or 5-10x more software that's useful to me, or that the software I'm using is suddenly working 5-10x better, etc.

However, I'm also struggling to see the supposed 5-10x'ing anywhere else. If power users are experiencing this much improvement, what projects were enabled by it?

Previously, I'd assumed I didn't know just because I'm living under a rock. So I've tried to get Deep Research to fetch me an overview, and it... also struggled to find anything concrete. Judge for yourself: one, two. The COBOL refactor counts, but that's about it. (Maybe I'm bad at prompting it?)

Even the AGI labs' customer-facing offerings aren't an endless trove of rich features for interfacing with their LLMs in sophisticated ways – even though you'd assume there'd be an unusual concentration of power users there. You have a dialogue box and can upload PDFs to it, that's about it. You can't get the LLM to interface with an ever-growing list of arbitrary software and data types, there isn't an endless list of QoL features that you can turn on/off on demand, etc.[1]

So I'm asking LW now: What's the real-world impact? What projects/advancements exist now that wouldn't have existed without LLMs? And if none of that is publicly attributed to LLMs, what pr...
Validating against a misalignment detector is very different to training against one
by mattmacdermott — 2025-03-04T15:41:04.692Z

Consider the following scenario:

We have ideas for training aligned AI, but they’re mostly bad: 90% of the time, if we train an AI using a random idea from our list, it will be misaligned.
We have a pretty good alignment test we can run: 90% of aligned AIs will pass the test and 90% of misaligned AIs will fail (for AIs trained using our list of ideas).
The alignment test is easy to circumvent: any misaligned model that fails the test is very close in weight-space to a misaligned model that passes the test (and any aligned model that fails the test is very close to an aligned model that passes it).

A naive alignment strategy would be to add a penalty term to our loss function that penalises failing the alignment test.

Question: If we apply this strategy and get an AI that passes the test, what’s the probability that it's aligned?

a) 50%. Just apply Bayes' rule to get P(aligned | passes test). The prior odds of aligned:misaligned are 10:90, the likehood ratio is 90:10, so we get posterior odds of 900:900, i.e. a probability of 50%.
b) 10%. We get the base rate P(aligned), because the misalignment test is trivially circumventable.

Answer:

Okay, it's not quite a mathematical question -- we can't be sure how the change to the loss landscape affects the path followed by SGD.

That said, it seems like it's probably b) 10%. Intuitively it seems like the only thing that will change is that if the model would have been detectably misaligned, now it will be undectably misaligned instead.

Certainly (a) is a naive and mistaken way to reason about what will happen -- we're not randomly sampling models that pass the alignment test.

New strategy: instead of adding a penalty term, we save the alignment test for the end of training. If the model passes, then great. If it fails, we throw the model away and train a new one with another random idea from our list. Repeat the process until we get an AI that passes the alignment test.

Question: What's the probability we get an aligned AI using this strategy?
a) 50%. It’s different this time, for some reason. 🤞

b) 10% again. Optimisation pressure is optimisation pressure; we’re just applying it in a different way.

Answer:

This time it follows mathematically from the setup that it’s (a) 50%. We are indeed randomly sampling models that pass the t

...
For scheming, we should first focus on detection and then on prevention
by Marius Hobbhahn — 2025-03-04T15:22:06.105Z

This is a personal post and does not necessarily reflect the opinion of other members of Apollo Research.

If we want to argue that the risk of harm from scheming in an AI system is low, we could, among others, make the following arguments:

Detection: If our AI system is scheming, we have good reasons to believe that we would be able to detect it.
Prevention: We have good reasons to believe that our AI system has a low scheming propensity or that we could stop scheming actions before they cause harm.

In this brief post, I argue why we should first prioritize detection over prevention, assuming you cannot pursue both at the same time, e.g. due to limited resources. In short, a) early on, the information value is more important than risk reduction because current models are unlikely to cause big harm but we can already learn a lot from them, and b) attempting prevention without good detection might lead to sneaky scheming which is potentially worse than doing nothing at all.

Techniques

Naturally, there is a lot of overlap between detection and prevention, e.g. a monitoring-based control approach would likely increase both detection and prevention. My guess is that at least 50% of things we could do to address scheming meaningfully increase detection and prevention. In these cases, we should obviously just do both.

However, some methods don’t overlap or emphasize different parts.

Detection: Prioritize methods that enable us to detect when the model is scheming.

Extensive evals: We design a wide range of scheming capability and propensity evaluations to get a prior expectation for how likely our AI system is scheming. In these evals, we could, for example, give the AI system a task and a set of constraints. In these scenarios, it is possible to achieve the task more efficiently by breaking some of the constraints. However, since the AI knows that it is not supposed to break the constraints, it has to do so covertly and not get caught if it wants to achieve the task efficiently. We then test how often the AI chooses to scheme, e.g. circumvent the constraints and hide the evidence. These evaluations could be capability evals like

Meinke et al., 2024

and

Benton et al., 2024

or more propensity-flavored.
Extensive deployment monitoring: During deployment, we constantly monitor the CoT and actions of the model. We monitor both for sc

...
Progress links and short notes, 2025-03-03
by jasoncrawford — 2025-03-04T15:20:35.619Z
Link: https://newsletter.rootsofprogress.org/p/links-and-short-notes-2025-03-03

Much of this content originated on social media. To follow news and announcements in a more timely fashion, follow me on Twitter, Notes, Farcaster, Bluesky, or Threads.

An occasional reminder: I write my blog/newsletter as part of my job running the Roots of Progress Institute (RPI). RPI is a nonprofit, supported by your subscriptions and donations. If you enjoy my writing, or appreciate programs like our fellowship and conference, consider making a donation. (To those who already donate, thank you for making this possible!)

Contents

Progress Conference 2025
Are you teaching progress at university?
A progress talk for high schoolers
Job opportunities
Fellowship opportunities
Project opportunities
Events
Writing announcements
Fund announcements
AI news
Energy news
Bio news
Queries

For paid subscribers:

A positive supply shock for truth
Elon is perpetually in wartime mode
The hinge of history
More quotes
AI doing things
RPI fellows doing things
Things you might want to read
Aerospace
Comments I liked
Politics
Fun

Progress Conference 2025

Save the date: Progress Conference 2025 will be October 16–19 in Berkeley, CA. Hosted by us, the Roots of Progress Institute, together with the Abundance Institute, the Foresight Institute, the Foundation for American Innovation, HumanProgress.org, the Institute for Humane Studies, and Works in Progress magazine. Speakers and more details to be announced this spring.

Progress Conference 2024 was a blast: Fantastic people, enchanting venue, great energy. Several people called it the best conference they had ever attended, full stop. (!) 2025 is going to be bigger and better!

Are you teaching progress at university?

Professors: are you teaching a “progress studies” course now/soon, or considering it?

I’ve heard from a few folks recently who are doing this. It might be useful to share syllabi and generally help each other out. We can serve as a hub for this! Reply and let me know.

A progress talk for high schoolers

I gave a talk to high schoolers:

The Future of Humanity—And How You Can Help

We hear a lot about disaster scenarios, from pandemic diseases to catastrophic climate change. Is humanity doomed? Or can we solve these challenges, and even create a future that is better than the world has ever seen? I make the case for problem-solving, based on both history and theory, and conclude by pointing toward

...
Formation Research: Organisation Overview
by alamerton — 2025-03-04T15:03:33.196Z

Thank you to Adam Jones, Lukas Finnveden, Jess Riedel, Tianyi (Alex) Qiu, Aaron Scher, Nandi Schoots, Fin Moorhouse, and others for the conversations and feedback that helped me synthesise these ideas and create this post.

Epistemic Status

: my own thoughts and research after thinking about lock-in and having conversations with people for a few months. This post summarises my thinking about lock-in risk.

TL;DR

This post gives an overview to Formation Research, an early stage nonprofit startup research organisation working on lock-in risk, a neglected category of AI risk.

Introduction

I spent the last few months of my master’s degree working with Adam Jones on starting a new organisation focusing on a neglected area of AI safety he identified. I ended up focusing on a category of risks I call ‘lock-in risks’. In this post, I outline what we mean by lock-in risks, some of the existing work regarding lock-in, the threat models we identified as being relevant to lock-in, and the interventions we are working on to reduce lock-in risks[1].

What Lock-In Means

We define lock-in risks as the probabilities of situations where features of the world, typically negative elements of human culture, are made stable for a long period of time.

The things we care most about when wanting to minimise lock-in risks (from our point of view as humans) are not all the features of the world, but the features of human culture, namely values and ethics, power and decision-making structures, cultural norms and ideologies, and scientific and technological progress. From the perspective of minimising risk, we are therefore most interested in potential lock-in scenarios that would be harmful, oppressive, persistent, or widespread.

While the definition encompasses lock-ins not related to AI systems, I anticipate that AI will be the key technology in their potential future manifestation. Therefore, the focus of the organisation is on lock-in risks in general, but we expect this to mostly consist of AI lock-in risks.

For example – North Korea under the Kim Family

Example of a totalitarian dictatorship under the Kim family since 1948
Centralised government and ideological control
Extensive surveillance, forced labour, and exploitation

Or, a generally intelligent autonomous system pursuing a goal and preventing interference

Example of a situation which could remain stable for a long time
Digital

...
On Writing #1
by Zvi — 2025-03-04T13:30:06.103Z

This isn’t primarily about how I write. It’s about how other people write, and what advice they give on how to write, and how I react to and relate to that advice.

I’ve been collecting those notes for a while. I figured I would share.

At some point in the future, I’ll talk more about my own process – my guess is that what I do very much wouldn’t work for most people, but would be excellent for some.

Table of Contents

How Marc Andreessen Writes.

How Sarah Constantin Writes.

How Paul Graham Writes.

How Patrick McKenzie Writes.

How Tim Urban Writes.

How Visakan Veerasamy Writes.

How Matt Yglesias Writes.

How JRR Tolkien Wrote.

How Roon Wants Us to Write.

When To Write the Headline.

Do Not Write Self-Deprecating Descriptions of Your Posts.

Do Not Write a Book.

Write Like No One Else is Reading.

Letting the AI Write For You.

Being Matt Levine.

The Case for Italics.

Getting Paid.

Having Impact.

How Marc Andreessen Writes

Marc Andreessen starts with an outline, written as quickly as possible, often using bullet points.

David Perell: When Marc Andreessen is ready to write something, he makes an outline as fast as possible.

Bullet points are fine. His goal is to splatter the page with ideas while his mind is buzzing. Only later does he think about organizing what he’s written.

He says: “I’m trying to get all the points out and I don’t want to slow down the process by turning them all into prose. It’s not a detailed outline like something a novelist would have. It’s basically bullet points.”

Marc is saying that first you write out your points and conclusion, then you fill in the details. He wants to figure it all out while his mind is buzzing, then justify it later.

Whereas I learn what I think as I write out my ideas in detail. I would say that if you are only jotting down bullet points, you do not yet know what you think.

Where we both agree is that of course you should write notes to remember key new ideas, and also that the organizing what goes where can be done later.

I do not think it is a coincidence that this is the opposite of my procedure. Yes, I have some idea of what I’m setting out to write, but it takes form as I write it, and as I write I understand.

If you’re starting with a conclusion, then writing an outline, and writing them quickly, that says you are looking to communicate what you already know, rather than seeking to yourself learn via t...
The Semi-Rational Militar Firefighter
by P. João — 2025-03-04T12:23:37.253Z

LessWrong Context:

I didn’t want to write this.

Not for lack of courage—I’d meme-storm Putin’s Instagram if given half a chance. But why?

Too personal.
My stories are tropical chaos: I survived the Brazilian BOPE (think Marine Corps training, but post-COVID).
I’m dyslexic, writing in English (a crime against Grice).
This is LessWrong, not some Deep Web Reddit thread.

Okay, maybe a little lack of courage.

And yet, something can be extracted from all this madness, right?

Then comes someone named Gwern. He completely ignores my thesis and simply asks:
"Tell military firefighter stories."

My first instinct was to dismiss him as an oddball—until a friend told me I was dealing with a legend of rationality. I have to admit: I nearly shit myself. His comment got more likes than the post I’d spent years working on.

Someone with, what, a 152 IQ wanted my accounts of surviving bureaucratic military hell? And I’m the same guy who applies scientific rigor to Pokémon analysis?

I didn’t want to expose my ass in LessWrong, but here we are. So, I decided to grant his request with a story that blends military rigidity with... well, whatever it is I do—though the result might be closer to Captain Caveman than Sun Tzu.

Firefighter Context:

Brazilian military firefighters are first and foremost soldiers. Their training is built on four pillars: first aid, rescue, firefighting, and aquatic survival.

We were in the jungle, undergoing a rescue training exercise with no food, alongside the BOPE—Brazil’s elite force, notorious for their grueling training and for carrying a skull-and-dagger emblem. Wherever they go, they shout their motto:
“Knife in the skull!”

The Knife:

After a week without food, they released animals into the jungle. The female recruits had to hunt, and they managed to kill a rabbit with a single clubbing blow—its eye popped out. Then they turned to me:

“Brito! Are you ‘knife in the skull?’”
“I’m knife in the hose, sir!”
“But… doesn’t a knife in the hose puncture the hose?”
“And doesn’t a knife in the skull puncture the skull?”
(Some laughter)
“Then prove you’re ‘knife’ and eat this rabbit’s eye raw!”

So, channeling the most primal, savage creature I knew, I swallowed the eye and croaked out: “My preciousssss!” —Smigle from The Lord of the Rings—

Later, during formation, another superior addressed my squad:
“We need more firefighters like this—who throw their whole bodies into following ...