trentmkelly/LessWrong-43k huggingface dataset:
"Long" timelines to advanced AI have gotten crazy short
by Matrice Jacobine — 2025-04-03T22:46:39.416Z
Link: https://helentoner.substack.com/p/long-timelines-to-advanced-ai-have

First post of @Helen Toner (of OpenAI board crisis fame)'s new Substack

It used to be a bold claim, requiring strong evidence, to argue that we might see anything like human-level AI any time in the first half of the 21st century. This 2016 post, for instance, spends 8,500 words justifying the claim that there is a greater than 10% chance of advanced AI being developed by 2036.

(Arguments about timelines typically refer to “timelines to AGI,” but throughout this post I’ll mostly refer to “advanced AI” or “human-level AI” rather than “AGI.” In my view, “AGI” as a term of art tends to confuse more than it clarifies, since different experts use it in such different ways.1 So the fact that “human-level AI” sounds vaguer than “AGI” is a feature, not a bug—it naturally invites reactions of “human-level at what?” and “how are we measuring that?” and “is this even a meaningful bar?” and so on, which I think are totally appropriate questions as long as they’re not used to deny the overall trend towards smarter and more capable systems.)

Back in the dark days before ChatGPT, proponents of “short timelines” argued there was a real chance that extremely advanced AI systems would be developed within our lifetimes—perhaps as soon as within 10 or 20 years. If so, the argument continued, then we should obviously start preparing—investing in AI safety research, building international consensus around what kinds of AI systems are too dangerous to build or deploy, beefing up the security of companies developing the most advanced systems so adversaries couldn’t steal them, and so on. These preparations could take years or decades, the argument went, so we should get to work right away.

Opponents with “long timelines” would counter that, in fact, there was no evidence that AI was going to get very advanced any time soon (say, any time in the next 30 years).2 We should thus ignore any concerns associated with advanced AI and focus instead on the here-and-now problems associated with much less sophisticated systems, such as bias, surveillance, and poor labor conditions. Depending on the disposition of the speaker, problems from AGI might be banished forever as “science fiction” or simply relegated to the later bucket.

Whoever you think was right, for the purposes of this post I want to point out that this debate made sense. “This enormously consequential technology might be built withi

...
An idea for avoiding neuralese architectures
by Knight Lee — 2025-04-03T22:23:21.653Z

One downside of an English chain-of-thought, is that each token contains only ≈17 bits of information, creating a tight information bottleneck.

Don't take my word for it, look at this section from a story by Daniel Kokotajlo, Thomas Larsen, elifland, Scott Alexander, Jonas V, romeo:

[...] One such breakthrough is augmenting the AI’s text-based scratchpad (chain of thought) with a higher-bandwidth thought process (neuralese recurrence and memory). [...]

Neuralese recurrence and memory

Neuralese recurrence and memory allows AI models to reason for a longer time without having to write down those thoughts as text.

Imagine being a human with short-term memory loss, such that you need to constantly write down your thoughts on paper so that in a few minutes you know what’s going on. Slowly and painfully you could make progress at solving math problems, writing code, etc., but it would be much easier if you could directly remember your thoughts without having to write them down and then read them. This is what neuralese recurrence and memory bring to AI models.

In more technical terms:

Traditional attention mechanisms allow later forward passes in a model to see intermediate activations of the model for previous tokens. However, the only information that they can pass backwards (from later layers to earlier layers) is through tokens. This means that if a traditional large language model (LLM, e.g. the GPT series of models) wants to do any chain of reasoning that takes more serial operations than the number of layers in the model, the model is forced to put information in tokens which it can then pass back into itself. But this is hugely limiting—the tokens can only store a tiny amount of information. Suppose that an LLM has a vocab size of ~100,000, then each token contains

log2(100k)=16.6

bits of information, around the size of a single floating point number (assuming training in FP16). Meanwhile, residual streams—used to pass information between layers in an LLM—contain thousands of floating point numbers.

One can avoid this bottleneck by using neuralese: passing an LLM’s residual stream (which consists of several-thousand-dimensional vectors) back to the early layers of the model, giving it a high-dimensional chain of thought, potentially transmitting over 1,000 times more information.

![](/hao2024-nowatermark-inexpandable.png)

Figure from Hao et

...
LessWrong merch?
by Brendan Long — 2025-04-03T21:51:47.190Z

Does the LessWrong team sell any merch, or have you considered doing it? Things like shirts, stickers, cute shoggoth plushies?

The reason I'm asking is that branded merch would be an easy way to let people find each other for rationalist meetups. If you see someone wearing a LessWrong shirt, you're probably in the right place[1].

I'm thinking of just bringing one of the books and leaving it on the table, but it seemed worth asking if there's anything else, especially since Lightcone seems to need money (who knows, maybe you'll sell 10,000 plushies and solve all of your funding problems!).

^

This is inspired by a local meetup, where the organizer always has a giant stuffed Pikachu with her.
Open Challenges in Representation Engineering
by j_we — 2025-04-03T19:21:45.971Z

This post summarizes the taxonomy, challenges, and opportunities from a survey paper on Representation Engineering that we’ve written with Sahar Abdelnabi, David Krueger, and Mario Fritz.

If you’re familiar with RepE feel free to skip to the “Challenges” and “Opportunities” sections.

What is Representation Engineering?

Representation Engineering (RepE) is a class of techniques that manipulate the representations of a concept in order to control its behaviour with regard to that concept. To achieve this, they (1) identify how the targeted concept is represented within the model and (2) use that information to steer the model’s representations on new inputs.

Representation Engineering first identifies how a concept is represented in the activation space of the model and then steers that representation to control the model’s behavior.

RepE includes methods such as Activation Steering e.g. Contrastive Activation Addition (CAA), Low-Rank Representation Adaptation (LoRRA), steering with Sparse Autoencoders, Bi-directional Preference Optimization (BiPO).

By tapping into models’ representations, RepE offers two main advantages: (1) Improve understanding: RepE identifies how human-understandable concepts are represented in the models’ activation space; steering that representation can verify whether the representation has the expected influence on the outputs and can offer an easy way to understand the model's internals. (2) Control: RepE is promising as a powerful tool to control the behaviour, personality, encoded beliefs, and performance of an LLM, allowing us to make models behave in safe and desired ways. Since no training is required, RepE can be cheaper, more data efficient, and more flexible to different users and situations than other methods while causing less deterioration to the model’s performance

RepE techniques have been applied to prevent harmful outputs, make models more truthful, align them with human preferences, change the model's situational beliefs, induce it to carry out specific tasks, change its goals, improve reasoning performance and generally interpret model internals.

Our Taxonomy

A pipeline to perform Representation Engineering has three aspects. We use CAA to illustrate them:

Representation Identification aims to find out how the concept is represented in the LLM’s activations. Such methods can calculate the difference in activations from specifi

...
Why Have Sentence Lengths Decreased?
by Arjun Panickssery — 2025-04-03T17:50:29.962Z
Link: https://arjunpanickssery.substack.com/p/why-have-sentence-lengths-decreased

“In the loveliest town of all, where the houses were white and high and the elms trees were green and higher than the houses, where the front yards were wide and pleasant and the back yards were bushy and worth finding out about, where the streets sloped down to the stream and the stream flowed quietly under the bridge, where the lawns ended in orchards and the orchards ended in fields and the fields ended in pastures and the pastures climbed the hill and disappeared over the top toward the wonderful wide sky, in this loveliest of all towns Stuart stopped to get a drink of sarsaparilla.”
— 107-word sentence from Stuart Little (1945)

Sentence lengths have declined. The average sentence length was

49 for Chaucer

(died 1400),

50 for Spenser

(died 1599),

42 for Austen

(died 1817),

20 for Dickens

(died 1870),

21 for Emerson

(died 1882),

14 for D.H. Lawrence

(died 1930), and

18 for Steinbeck

(died 1968).

J.K Rowling averaged 12

words per sentence (wps) writing the Harry Potter books 25 years ago.

So the decline predates television, the radio, and the telegraph—it’s been going on for centuries. The average sentence length in newspapers

fell from 35wps to 20wps

between 1700 and 2000. The presidential State of the Union address has gone

from 40wps down to under 20wps

, and the inaugural addresses had a similar decline. (From Jefferson through T. Roosevelt, the SOTU address was delivered to Congress without any speech, and print was the main way that inaugural addresses were consumed for most of their history.) Warren Buffett’s annual letter to shareholders

dropped from 17.4wps to 13.4wps

between 1974 and 2013.

SlateStarCodex’s

ten recommended blog posts

have 22wps. My own

top 10 posts

have 20wps. Even

top medical journals have under 25wps

. The FAA, the European Commission, and various legal institutions have style guides recommending to stay under 20wps. Skimming r/writing, it looks like people recommend 10-15wps for fiction (

HPMOR

has 15wps). It’s possible that sentence lengths will stop declining only when we hit a physical limit on how short sentences can reasonably become. The best-selling hardboiled novella The Postman Always Rings Twice (1934) has 11wps, while I saw one source claiming that Jurassic Park (1990) has only 9wps.

Several explanations present themselves for why sentence lengths have decreased. They aren’t mutually exclusive; it could be that all of them c...
Announcing ILIAD2: ODYSSEY
by Alexander Gietelink Oldenziel — 2025-04-03T17:01:06.004Z

We are pleased to announce ILIAD2: ODYSSEY—a 5-day conference bringing together 100+ researchers to build scientific foundations for AI alignment. This is the 2nd iteration of ILIAD, which was

first held in the summer of 2024

.

***

Apply to attend by June 1!

***

When: Aug 25-29, 2025
Where: @Lighthaven (Berkeley, US)
What: Unconference with participant-led programming, 100+ attendees.
Who: Currently confirmed attendees include Scott Garrabrant, Daniel Murfet, James Crutchfield, Jesse Hoogland, Adam Shai, and Jacob Hilton
Costs: Tickets are free. Financial support for travel and accommodations is available on a needs basis.

See our website here. For any questions, email iliadconference@gmail.com

About ODYSSEY

ODYSSEY is a 100+ person conference about alignment with a mathematical focus. ODYSSEY will feature an unconference format—meaning that participants can propose and lead their own sessions. We believe that this is the best way to release the latent creative energies in everyone attending.

The ILIAD conferences are the premier gathering for all things theory and alignment. Topics include: Singular Learning Theory, Agent Foundations, Causal Incentives, Computational Mechanics, Safety-by-Debate, Scalable Oversight and more.

To get a better sense of what might happen, you can view the schedule for last year’s ILIAD conference

here

.

Financial Support

Financial support for accommodation & travel are available on a needs basis. We may not be able to offer support to everyone who requests it.

Proceedings

We are planning to announce a call for papers for the ODYSSEY Proceedings. Submissions will undergo a peer review process, and provisionally accepted papers will be invited to give a featured talk at ODYSSEY. If you would like to submit a manuscript to the Proceedings, please indicate so your application form and we will be in touch. The submission deadline will be around the end of June. More details to be announced.

Artwork

We are looking for Odyssey-themed artwork to use for ODYSSEY. You can share suggestions in the comments below or send us a message at iliadconference@gmail.com.
I "invented" semimeasure theory and all I got was imprecise probability theory
by Cole Wyeth — 2025-04-03T16:33:21.820Z

Epistemic status: Informal summary of / intuition behind formal research (developing semimeasure theory conjectures from IUAI with Marcus Hutter). Paper is forthcoming. Partially supported by the LTFF.

This is post is a rather philosophical and intuitive take on Solomonoff induction and model misspecification[1]. It's also a misadventure through the mathematical landscape. And so it begins with a call to adventure, which in this case is me getting nerd-sniped by a minor technical difference between two definitions of the value function in (general history-based) reinforcement learning.

The original definition is the iterative value function, which is (with policy π, environment ν):

Vit,mνπ=∑a1:m,e1:mνπ(ae1:m)∑mt=1γtrt

Refresher on history-based RL notation. Here νπ is the distribution on the history of action a and percepts e induced by the interaction of environment ν and policy π. Also, the rewards are part of the percepts: et:=otrt, γt is a discount factor which should sum to ≤1, and m is the lifetime of the agent (or the horizon).

This makes perfect sense when you are using proper probability distributions for your policy and environment, as a sane person would do. Unfortunately, in algorithmic information theory we must use Hutter's universal chronological semimeasure, which is defective - all of its conditionals fail to sum to 1 because the percept sequence can end early (for instance, before time m). This is sometimes interpreted as "death," which I'll take issue with later, but for now the point is that you want to give an agent credit for the rewards it earns before the percept sequence ends, and the iterative value function doesn't do that.

The recursive value function is preferable for the semimeasure case:

Vrec,mνπ=∑mt=1∑a1:t,e1:tγtrtνπ(ae1:t)

These are equivalent when everything is a sane probability measure.

Normally, we send m→∞. One way to think of this is that we're taking an integral instead of a sum. The integrand is ∑∞t=1γtrt, which is a nicer view if we want to consider other utility functions. The iterative value function is analogous to a Riemann-Stieltjes integral, because it basically slices its domain up more and more finely (this can be made precise, and is easiest to see if you visualize ae as a binary expansion i...
AI 2027: What Superintelligence Looks Like
by Daniel Kokotajlo — 2025-04-03T16:23:44.619Z
Link: https://ai-2027.com/

In 2021 I wrote what became my most popular blog post: What 2026 Looks Like. I intended to keep writing predictions all the way to AGI and beyond, but chickened out and just published up till 2026.
Well, it's finally time. I'm back, and this time I have a team with me: the AI Futures Project. We've written a concrete scenario of what we think the future of AI will look like. We are highly uncertain, of course, but we hope this story will rhyme with reality enough to help us all prepare for what's ahead.

You really should go read it on the website instead of here, it's much better. There's a sliding dashboard that updates the stats as you scroll through the scenario!

But I've nevertheless copied the first half of the story below. I look forward to reading your comments.

Mid 2025: Stumbling Agents

The world sees its first glimpse of AI agents.

Advertisements for computer-using agents emphasize the term “personal assistant”: you can prompt them with tasks like “order me a burrito on DoorDash” or “open my budget spreadsheet and sum this month’s expenses.” They will check in with you as needed: for example, to ask you to confirm purchases.[1] Though more advanced than previous iterations like Operator, they struggle to get widespread usage.[2]

Meanwhile, out of public focus, more specialized coding and research agents are beginning to transform their professions.

The AIs of 2024 could follow specific instructions: they could turn bullet points into emails, and simple requests into working code. In 2025, AIs function more like employees. Coding AIs increasingly look like autonomous agents rather than mere assistants: taking instructions via Slack or Teams and making substantial code changes on their own, sometimes saving hours or even days.[3] Research agents spend half an hour scouring the Internet to answer your question.

The agents are impressive in theory (and in cherry-picked examples), but in practice unreliable. AI twitter is full of stories about tasks bungled in some particularly hilarious way. The better agents are also expensive; you get what you pay for, and the best performance costs hundreds of dollars a month.[4] Still, many companies find ways to fit AI agents into their workflows.[5]

Late 2025: The World’s Most Expensive AI

OpenBrain is building the biggest datacenters the world has ever seen.[6]

(To avoid singling out any one existing com...
Consider Breathwork
by Jonathan Moregård — 2025-04-03T14:46:57.398Z
Link: https://honestliving.substack.com/p/consider-breathwork

Breathing patterns influence cognition. A guided 20-minute breathwork session can shift your mental state—calming anxiety, boosting energy, and helping you reorient when stuck on a problem.[1] This post shares a personal account of its effects, practical instructions, guided sessions, and key safety notes.

I'm sitting in my room feeling calm, energetic and present. Pleasurable tingles pulse down my arms and into my fingers. My mind is quiet, unhurried and joyful. I just finished

20 minutes of guided breathwork

, and I’m amazed by how immediate and profound the effects are.

Breathwork has become one of my go-to tools, both when I'm anxious and when I'm sluggish. It’s rare to find something that both calms and energizes, bringing presence while also inviting action. Breathwork manages this combo. It’s like the energy of coffee without the jitteriness, combined with the relaxation of lying on a warm beach—without the sweat, drowsiness, and sunburn.

Besides these effects, breathwork acts as a reset for my thinking patterns. When I get stuck on a problem, it's often because of invisible assumptions or rigid framings. Taking a break—like sleeping on it—usually helps. But breathwork is quicker! After a session, I often come back able to see the problem from a new angle. I’m curious if others can do the same—using breathwork for quicker iteration on difficult “deep work” challenges.

If you’ve never tried it, I suggest using

20 minutes

to try it out. If you like the effects, consider trying it out daily for a week or so—enough time to build some familiarity,

reduce friction

and make it part of your toolkit.

Disclaimer: While breathwork is generally safe, in rare cases people faint—so don’t practice near/in water[2] or while driving. Also, breathwork is risky for pregnant people, epileptics, and people with cardiovascular issues or other serious health concerns. Don’t push yourself, and talk to a medical professional if uncertain.

My favorite breathwork style is the Wim Hof method. It involves cycles of deep breathing followed by breath holds.

It’s easiest to do while lying down. Breathing can be done through the nose, mouth, or a combination. You breathe into your belly at first, expanding into your chest without pause. The outbreaths should be effortless, just relaxing and letting the air out.

I’ll leave two guided sessions below for you to try out. See what ...
AI #110: Of Course You Know…
by Zvi — 2025-04-03T13:10:05.674Z

Yeah. That happened yesterday. This is real life.

I know we have to ensure no one notices Gemini 2.5 Pro, but this is rediculous.

That’s what I get for trying to go on vacation to Costa Rica, I suppose.

I debated waiting for the market to open to learn more. But f*** it, we ball.

Table of Contents

Also this week: More Fun With GPT-4o Image Generation, OpenAI #12: Battle of the Board Redux and Gemini 2.5 Pro is the New SoTA.

The New Tariffs Are How America Loses. This is somehow real life.

Is AI Now Impacting the Global Economy Bigly? Asking the wrong questions.

Language Models Offer Mundane Utility. Is it good enough for your inbox yet?

Language Models Don’t Offer Mundane Utility. Why learn when you can vibe?

Huh, Upgrades. GPT-4o, Gemini 2.5 Pro, and we partly have Alexa+.

On Your Marks. Introducing PaperBench. Yes, that’s where we are now.

Choose Your Fighter. How good is ChatGPT getting?

Jevons Paradox Strikes Again. Compute demand is going to keep going up.

Deepfaketown and Botpocalypse Soon. The only answer to a bad guy with a bot.

They Took Our Jobs. No, AI is not why you’ll lose your job in the short term.

Get Involved. Fellowships, and the UK AISI is hiring.

Introducing. Zapier releases its MCP server, OpenAI launches AI Academy.

In Other AI News. Google DeepMind shares 145 page paper, but no model card.

Show Me the Money. The adventures of the efficient market hypothesis.

Quiet Speculations. Military experts debate AGI’s impact on warfare.

The Quest for Sane Regulations. At what point do you just give up?

Don’t Maim Me Bro. Further skepticism that the MAIM assumptions hold.

The Week in Audio. Patel on Hard Fork, Epoch employees debate timelines.

Rhetorical Innovation. As usual it’s not going great out there.

Expect the Unexpected. What are you confident AI won’t be able to do?

Open Weights Are Unsafe and Nothing Can Fix This. Oh no, OpenAI.

Anthropic Modifies its Responsible Scaling Policy. Some small changes.

If You’re Not Going to Take This Seriously. I’d prefer if you did?

Aligning a Smarter Than Human Intelligence is Difficult. Debating SAEs.

Trust the Process. Be careful exactly how much and what ways.

People Are Worried About AI Killing Everyone. Elon Musk again in brief.

The Lighter Side. Surely you’re joking, Mr. Human. Somehow.

The New Tariffs Are H

...