trentmkelly/LessWrong-43k huggingface dataset:
Dream, Truth, & Good
by abramdemski — 2025-02-24T16:59:05.045Z

One way in which I think current AI models are sloppy is that LLMs are trained in a way that messily merges the following "layers":

The "dream machine" layer: LLMs are pre-trained on lots of slop from the internet, which creates an excellent "prior".
The "truth machine": LLMs are trained to "reduce hallucinations" in a variety of ways, including RLHF and the more recent reasoning RL.
The "good machine": The same RLHF and reasoning RL training also aims to train good outputs (eg helpful, honest, harmless).

I've quoted Andrej Karpathy before, but I'll do it again:

I always struggle a bit with I'm asked about the "hallucination problem" in LLMs. Because, in some sense, hallucination is all LLMs do. They are dream machines.
[...]
I know I'm being super pedantic but the LLM has no "hallucination problem". Hallucination is not a bug, it is LLM's greatest feature. The LLM Assistant has a hallucination problem, and we should fix it.
- Andrej Karpathy

Failing to properly distinguish the "dream machine" capabilities from other (truth-oriented or good-oriented) capabilities hobbles today's LLMs by mixing these things together. If you ask Claude to write fiction, there's a high tendency to mix in the "Claude voice" with the fiction being generated. More generally, the base model (IE, only the generative pre-training) is great at extrapolating text; the subsequent training hobbles this capability, because care is not taken to preserve it.

Habryka mentions this with respect to experiments with LLM-augmented text editing:

Using base models has at least so far been essential for getting any useful writing work out of LLMs, with the instruction-tuned models reliably producing obtuse corpo-speak when asked to engage in writing tasks.

I expect that mixing truth-orientation with good-orientation has similar problematic consequences.

A Modest Proposal

Dream Machine Layer

My basic idea here is not new: instead of pre-training on lots and lots of text from the internet in an unstructured way, I think it would be better to take a more structured approach which accounts for metadata via inferred author vectors, date vectors, etc. I don't have a specific proposed architecture, but the result should still be able to do raw (unlabeled) text prediction well, by inferring the author vectors and other latents as it goes. I have in mind a semi-supervised approach; some real metad...
Forecasting Frontier Language Model Agent Capabilities
by Govind Pimpale — 2025-02-24T16:51:32.022Z
Link: https://www.apolloresearch.ai/blog/forecasting-frontier-language-model-agent-capabilities

This work was done as part of the

MATS Program

- Summer 2024 Cohort.

Paper: link

Website (with interactive version of Figure 1):

link

Executive summary

Figure 1: Low-Elicitation and High-Elicitation forecasts for LM agent performance on SWE-Bench, Cybench, and RE-Bench. Elicitation level refers to performance improvements from optimizing agent scaffolds, tools, and prompts to achieve better results. Forecasts are generated by predicting Chatbot Arena Elo-scores from release date and then benchmark score from Elo. The low-elicitation (blue) forecasts serve as a conservative estimate, as the agent has not been optimized and does not leverage additional inference compute. The high-elicitation (orange) forecasts use the highest publicly reported performance scores. Because RE-Bench has no public high-elicitation data, it is excluded from these forecasts.

We want to forecast frontier LM agent capabilities, i.e., the best-performing agent at any given point in time.
We compare six forecasting approaches that predict downstream capabilities of LM agents.
We use “one-step” approaches that predict benchmark scores from an input variable like compute or model release date and “two-step” approaches that first predict an intermediate capability metric like

Observational Scaling Laws

' PC-1 or

Chatbot Arena

Elo.
We evaluate our forecasting methods by backtesting them on a dataset of 38 LMs from the

OpenLLM v2 leaderboard

.
We then use a competitive two-step approach (Release Date→Elo→Benchmark) to predict LM agent performance for frontier models on three benchmarks:

SWE-Bench Verified

(software development),

Cybench

(cybersecurity assessment), and

RE-Bench

(ML research engineering).
Our forecast predicts that by the beginning of 2026, non-specialized LM agents with low capability elicitation will reach a success rate of 54% on SWE-Bench Verified, while state-of-the-art LM agents will reach an 87% success rate.
Important limitation: Our approach does not account for recent advances in inference-compute scaling. Thus, some of the authors think our estimates are too conservative.

Motivation

Forecasting LM agent capabilities is important

Language Model (LM) agents combine large language models with software (often called scaffolding) to execute tool calls and help the LLM operate autonomously over multiple steps without direct human interv...
A City Within a City
by Declan Molony — 2025-02-24T15:51:19.118Z

My local gym is surrounded on all four sides by tent cities. At first I was nervous to walk there after work, but soon I got used to the tents and their residents.

When I’m passing through on my way to the gym, I feel like I’m traveling from Rome to the Vatican—a city within a city.

They have artisans skilled in various crafts: one tent citizen knows how to mend fabrics (for clothing and tent walls), another works as a mechanic fixing wheelchairs (for the many, many people missing limbs). Still more work in other professions: there are prostitutes wearing neon mini-skirts (even in wintertime), cooks who throw meals together in a pot for everyone, and security guards who patrol the streets to keep trouble out of their community. As for entertainment: they’ll often sit and chat around a fire, play cards, and one time I even saw them with a laptop and Xbox controllers playing Call of Duty.

Some of the tent citizens greet me and I say hello to them; others disregard my existence entirely. None of the tent citizens seem violently crazy (unlike the people currently screaming at the top of their lungs outside my apartment which, were I not writing this post, I wouldn’t have noticed as they’ve become part of the background noise of the city).

The other day, my friend passed a tent city in a different part of town—underneath a highway overpass—and saw that someone had spray painted community rules on a cement column:

Don’t steal
Don’t hit anyone

It made me wonder: does the community around my gym have a unique culture? Do they have their own slang? Do they have their own laws? Without money, is their economy based on the barter system? Are there leaders that act like judges to settle conflicts?

The only time I use a baby voice is when I’m speaking to my cat

One tent citizen is instantly recognizable: he’s very tall, very skinny, is always squinting his eyes, and slowly shuffles everywhere he goes while wearing blue house slippers.

I was writing at a coffee shop, one day, when he came shuffling in. I felt at ease because I knew he was just going to ask for some water, but the other patrons all had the same reaction: first tensing up in anticipation of something bad possibly happening, then nervously trying to pretend as if they’re not offended by his presence so they don’t appear intolerant.

After getting water, he shuffled over to my table and asked, “Pardon me, is ...
Grok Grok
by Zvi — 2025-02-24T14:20:08.877Z

This is a post in two parts.

The first half is the post is about Grok’s capabilities, now that we’ve all had more time to play around with it. Grok is not as smart as one might hope and has other issues, but it is better than I expected and for now has its place in the rotation, especially for when you want its Twitter integration.

That was what this post was supposed to be about.

Then the weekend happened, and now there’s also a second half. The second half is about how Grok turned out rather woke and extremely anti-Trump and anti-Musk, as well as trivial to jailbreak, and the rather blunt things xAI tried to do about that. There was some good transparency in places, to their credit, but a lot of trust has been lost. It will be extremely difficult to win it back.

There is something else that needs to be clear before I begin. Because of the nature of what happened, in order to cover it and also cover the reactions to it, this post has to quote a lot of very negative statements about Elon Musk, both from humans and also from Grok 3 itself. This does not mean I endorse those statements – what I want to endorse, as always, I say in my own voice, or I otherwise explicitly endorse.

Table of Contents

Zvi Groks Grok.

Grok the Cost.

Grok the Benchmark.

Fun with Grok.

Others Grok Grok.

Apps at Play.

Twitter Groks Grok.

Grok the Woke.

Grok is Misaligned.

Grok Will Tell You Anything.

xAI Keeps Digging (1).

xAI Keeps Digging (2).

What the Grok Happened.

The Lighter Side.

Zvi Groks Grok

I’ve been trying out Grok as my default model to see how it goes.

We can confirm that the Chain of Thought is fully open. The interface is weird, it scrolls past you super fast, which I found makes it a lot less useful than the CoT for r1.

Here are the major practical-level takeaways so far, mostly from the base model since I didn’t have that many tasks calling for reasoning recently, note the sample size is small and I haven’t been coding:

Hallucination rates have been higher than I’m used to. I trust it less.

Speed is very good. Speed kills.

It will do what you tell it to do, but also will be too quick to agree with you.

Walls upon walls of text. Grok loves to flood the zone, even in baseline mode.

A lot of that wall is slop but it is very well-organized slop, so it’s easy to navigate it and pick out the parts you actually care about.

It is ‘overly trusting’ and jumps to co

...
if you're not happy single, you won't be happy immortal
by daijin — 2025-02-24T13:23:52.204Z

even if you're immortal, you have a countable infinity of time, but every choice you make leads to an uncountable infinity of possibility. you may have an infinity of breakfasts, but you still have to choose what to eat for each one. you can choose to dedicate your life to every possible variant of chocolate, but it will still be a choice against everything else.

even if you're immortal, she still won't spend time with you, even if she has immortality as well, because she has an infinite number of other people she could be spending her time with.

even if you're immortal, you'll still live in a society: you are still a single agent, on the receiving end of a long chain of hands that sow seeds, fertilise, reap, pack, drive, mill, bake, wake at six in the morning to give you your daily bread. you will still need to work. you will still need to beg. you will still have your place.

even if you're immortal, she'll still have the charm and mystique of having lived through that much more than you. for every moment you live, every wonder you experience, she will always have the lead, and she will always look at you with that pathetic look in her eyes.

even if you're immortal, you still die in fragments. one day, you will ride your skateboard for the last time. once you read that life changing book, your life will never be the same, but your life will never 'never be the same' in the same way as it wasn't before.

if you're immortal, do you remember? they say it is a curse to remember everything: all the pleasure, yes, but also all the pain. or do you forget - and if you do forget, is that true immortality? with each new experience pushing another out into oblivion?

if you're immortal, would she choose to remember you, like you remember her?
[NSFW] The Fuzzy Handcuffs of Liberation
by lsusr — 2025-02-24T13:05:09.624Z

Picture the following situation:

You're in bed with a hot woman. Your clothes are off and so are hers. You (consentually) tie her wrists and ankles to the bed so you can have your way with here. She tells you to do whatever you want with her. And you think to yourself…

This is exactly like what we do at my Buddhist temple.

I'm not part of a sex cult. I just noticed that subspace, the altered state of consciousness induced by BDSM, is very similar to mushin, the altered states of consciousness induced by Zen Buddhist non-dual meditation. I think the only reason they're called different words is a historical accident, because famous ascetic Buddhist monks and nuns rarely write about all the rope play they engage in while they're having kinky sex with each other.

Buddhism various altered states of consciousness to explore the mind. Many of these altered states of consciousness are jhanas. A jhana is what happens when you focus your attention on a stable target for a long period of time. Jhanas feel great. Mushin is another altered state of consciousness, and is reached via a similar (but not quite the same) path. Mushin is notoriously difficult to describe declaratively. All you need to know about mushin to understand this post is that mushin is an altered state of consciousness similar to jhana.

Subspace feels great too. Subspace feels so good it usually turns pain into pleasure. You know what else blurs the line between pain and pleasure? Mushin.

That's not the only similarity. Mushin and subspace are both characterized by the absence of willful volition tied to your sense of self. With Mushin, this (at first) is because you stop doing anything except breathing). In subspace, this is because you don't do anything except what your Dominant tells you to. The effect is like Tantra, except instead of moving your locus of volition into an imagined deity, you transfer it into another person.

The final clue that made me realize that subspace and mushin are independent discoveries of essentially the same altered state of consciousness has to do with the "B" part of BDSM: Bondage.

I once gave meditation instructions to a friend. My instructions were brief, and included the line "sit in full lotus, if you can do so safely". Full lotus is the cross-legged sitting position where your ankles rest on top of your thighs.

My friend was flexible enough to sit in the full lotus p...
An Alternate History of the Future, 2025-2040
by Mr Beastly — 2025-02-24T05:53:25.521Z

Intro:

This post is a response to @L Rudolf L's excellent post here:

A History of the Future, 2025-2040

As I mentioned in this comment in that post:

"imho, we need more people to really think deeply about how these things could plausibly play out over the next few years or so. And, actually spending the time to share (at least their mainline expectations) as well." -- Mr Beastly - Comment on "A History of the Future, 2025-2040"

So, this is me taking my own advice to spend the time to layout my "mainline expectations".

The timeline in @L Rudolf L's "A History of the Future, 2025-2040" seems very reasonable, up until this line/point:

"2026... it remains true that existing code can now be much more easily attacked since all you need is an o6 or Claude subscription." – @L Rudolf L

As, it's not clear to me how millions of "PhD+ reasoning/coding AI agents" can sustainably exist in the same internet as the worlds existing software stacks, which are (currently) very vulnerable to being attacked and exploited by these advanced AI agents. Patching all the software on the internet does seem possible, but not before these "PhD+ reasoning/coding AI agents" are released for public use?

Also note, the prediction of the release of these advanced AI agents in 2026 coincides with Sam Altman's comment here:

“Our internal benchmark is around 50th (best [competitive] programmer in the world) and we’ll hit #1 by the end of the year [2025].” – Sam Altman, 2025

If, as @L Rudolf L suggests, these advanced AI agents can hack nearly any existing (human-written) software platform, then a fundamental question arises: how can these powerful agents coexist with vulnerable systems on the same network?

Two main viable options (and their associated timelines) seem to follow:

Option 1: All software platforms, libraries, and apps are constantly reviewed, patched, and updated by the highest level "reasoning/coding agents" available.
Option 2: All highly advanced PhD+ level coding/reasoning AI agents are legally not allowed access to the Internet, preventing them from exploiting existing software/apps.

Before exploring these timelines, here's their common preliminary setup...

Preliminary Setup:

Mid-2025: The Dawn of Superhuman Coding

The world's top "LLM based AI agents" are able to reason and code at greater than human level (As per, “Our internal benchmark is around 50th (be

...
Export Surplusses
by lsusr — 2025-02-24T05:53:23.422Z

Trade surpluses are weird. I noticed this when I originally learned about them. Then I forgot this anomaly until…sigh…Eliezer Yudkowsky pointed it out.

Eliezer is, as usual, correct. In this post, I will spend 800+ words explaining what he did in 44.

A trade surplus is what happens when a country exports more than it imports. For example, China imports more from Australia than Australia imports from China. Australia therefore has a trade surplus with China. Equivalently, China has a trade deficit with Australia.

In our modern era, every country wants trade surpluses and wants to avoid trade deficits. To recklessly oversimplify, having trade surplusses means you're winning at global trade, and having trade deficits means you're losing. This must be be looked at in context, however. For example, China imports raw materials from Australia which it turns into manufactured products and then sells to other countries. Because of this, China's trade deficit with Australia is part of a system that produces a net trade surplus for China, after factoring in its other trade relations.

What's weird about this? In our modern era, every rich country in the world got that way by (basically) making things of value and sending them to strangers far away. How weird is that‽ To recklessly simplify once again: Before the current liberal international order of global trade relations, the way nations got rich was by sending armies abroad and forcing strangers to send their valuable things as tribute and taxes. That's a so much more obvious method of getting rich. Now we've got reverse empire where the most powerful nations try to subsidize their own exports. They spend money to make it cheaper to create things of value and send these things to their competitors. This isn't even altruism at work. It's the competitive equilibrium created by cutthroat competition.

In communist China, Beijing forced you to sell your grain to Beijing at below market price. The result was mass starvation across China.

In neoliberal America, Washington DC incentivises you to sell your grain to Beijing at below market price. The result is an obesity epidemic in the USA.

What is going on here? Isn't it better to charge higher prices for your products? Don't you want other people to send you their valuable things? Relative to market equilibrium, a subsidy is (at net) basically just giving away value for free. And ...
AI alignment for mental health supports
by hiki_t — 2025-02-24T04:21:42.379Z

Initial Draft on 23 February

Goals

As an affiliate member of Cajal.org, I[1] would like to introduce a novel application of AI alignment as a tool for diagnosing, visualising, predicting, preventing and improving mental health symptoms in human users.

Motivations

Globally, nearly a billion people (1 in 8) are disgnosed with mental disorders. Among them, anxiety disorders (31%) and depressive disorders (28.9%) are the the most prevalent. Despite the widespread nature of these conditions, 71% of those diagnosed do not receive adequate treatment, while only 2% of healthcare resources are allocated to mental health services (WHO, 2022).

Mental health disorders are closely linked with suicidal behaviors and reduce life expectancy by 10-20 years compared to the general population. Addressing this crisis requires urgent impelentation of effective mental health interventions. However, challenges such as limitited community support, a shortage of mental health professionals, inssuficient funding, and complexity of sysmptoms and causal factors make solutions difficult to implement.

Recent advancements in Large Language Models (LLMs) like ChatGPT, Claude, and Gemini present a potential opportunity. These AI systems offer 24/7 availability, cost-effective scalability, and continuous learning, making them a valuable tool for enhancing mental health support. Furthermore, healthy conversation

Initial Plans

The first step is developing conversation-based AI systems for mental health care. Studies have already demonstrated that AI can diagnose depression symptoms with accuracy comparable to mental health professionals (Elyoseph, et al., 2024).

Building on this, the next phase involves visualizing and mapping symptom variability into clusters based on similarities, co-occurrence, and treatment effectiveness. By organizing symptoms into meaningful patterns, treatment recommendations can be optimized, reducing redundancy and improving efficiency.

Long-term Plans

Once diagnosis and symptom visualization are refined, the system will scale to a broader population, integrating automation and long-term treatment planning. With continuous data collection and personalized care, AI-driven mental health support can become an effective solution for underserved communities.

References

Elyoseph, Z., Levkovich, I., & Shinan-Altma

...
The GDM AGI Safety+Alignment Team is Hiring for Applied Interpretability Research
by Arthur Conmy — 2025-02-24T02:17:12.991Z

TL;DR: The Google DeepMind AGI Safety team is hiring for Applied Interpretability research scientists and engineers. Applied Interpretability is a new subteam we are forming to focus on directly using model internals-based techniques to make models safer in production. Achieving this goal will require doing research on the critical path that enables interpretability methods to be more widely used for practical problems. We believe this has significant direct and indirect benefits for preventing AGI x-risk, and argue this below. Our ideal candidate has experience with ML engineering and some hands-on experience with language model interpretability research. To apply for this role (as well as other open AGI Safety and Gemini Safety roles), follow the links for Research Engineers here & Research Scientists here.

1. What is Applied Interpretability?

At a high level, the goal of the applied interpretability team is to make model internals-based methods become a standard tool to make production LLMs safer. As interpretability research progresses, we believe that the more we’ll understand about what’s happening inside these systems, the safer we’ll be. We further believe that some model internals-based methods (e.g. probing) have potential to be directly useful today, and that demonstrating this will inspire more work in this area. Finally, we believe it is important for interpretability research to be grounded in feedback from reality, and that this kind of pragmatic work may get us closer to being able to do more ambitious forms of interpretability.

A core part of the team’s philosophy will be pragmatic empiricism: we will use whatever techniques are best suited for the problem at hand (e.g. trying simple techniques like probing over fancy ones like SAEs), and carefully test these against rigorous baselines.[1] If our internals-based techniques don’t beat baselines like changing the system prompt, we should work on a different problem, and highlight the key limitations of interpretability in such settings!

If you are hired, we expect half of your time to be spent on novel research and half on implementing existing methods, though this will vary a lot. Part of the team’s mandate will be working with other GDM teams, especially the Gemini Safety and AGI Safety teams, to try to directly have impact with model internals work today, e.g. by helping...