trentmkelly/LessWrong-43k huggingface dataset:
On GPT-4.5
by Zvi — 2025-03-03T13:40:05.843Z

It’s happening.

The question is, what is the it that is happening? An impressive progression of intelligence? An expensive, slow disappointment? Something else?

The evals we have available don’t help us that much here, even more than usual.

My tentative conclusion is it’s Secret Third Thing.

It’s a different form factor, with unique advantages, that is hard to describe precisely in words. It appears so far that GPT-4.5 has advantages in places like verbal intelligence, contextual adaptation, detailed knowledge, and a kind of abstract writing skill. It has better taste and aesthetics.

It is the first model I asked to help edit its own review, and it was (slightly) helpful.

It’s a different way to use a lot of compute to get more intelligence. When you need a reasoning model, or you need code written, do not call on GPT-4.5.

You need to carve out a new ‘place in your rotation’ for it.

Table of Contents

Introducing GPT-4.5.

The System Card.

Our Price Not Cheap.

Pay Up You Cheapskate.

While Supplies Last.

Benchmarks.

We Need Better Evaluations.

Positive Reactions.

Negative Reactions.

Predictions.

The Lighter Side.

Introducing GPT-4.5

Altman tells us up front: It’s a different kind of intelligence.

Sam Altman: GPT-4.5 is ready!

Good news: It is the first model that feels like talking to a thoughtful person to me. I have had several moments where I’ve sat back in my chair and been astonished at getting actually good advice from an AI.

Bad news: It is a giant, expensive model. We really wanted to launch it to Plus and Pro subscribers at the same time, but we’ve been growing rapidly and are out of GPUs. We will add tens of thousands of GPUs next week and roll it out to the Plus tier then. (Hundreds of thousands are coming soon, and I’m pretty sure you all will use every one we can accumulate.)

This isn’t how we want to operate, but it’s hard to perfectly predict growth surges that lead to GPU shortages.

A heads-up: This isn’t a reasoning model and won’t crush benchmarks. It’s a different kind of intelligence, and there’s a magic to it I haven’t felt before. I’m really excited for people to try it!

Kai: It was a *character-building* privilege to post-train GPT 4.5.

The official announcement is here.

The System Card

We get to be brief on this one. In terms of preparedness framework risk, this is actively a step down from the reasoning models, with Cybersecur...
Coalescence - Determinism In Ways We Care About
by vitaliya — 2025-03-03T13:20:44.408Z

(epistemic status: all models are wrong but some models are useful; I hope this is at least usefully wrong. also if someone's already done things like this please link me their work in the comments as it's very possible I'm reinventing the wheel)

I think utility functions are a non-useful frame for analysing LLMs; in this article I'll define a measure, coalescence, where approaching a coalescence of 1 can be qualitatively considered "acting more consistently"; a coalescence of 1 implies that an LLM is "semantically deterministic" even with non-deterministic syntax choice. Importantly, estimates of coalescence are computable, and I suspect correlate with a qualitative sense of "how well an LLM conforms to a utilitarian policy" (while making no guarantee of what that policy is). I end with evidence that a toy case on a real-world LLM where a decrease in the temperature parameter can result in an increase in semantic-level output randomness.

the relative disutility of utility

A common historical assumption was that if we ever made That Kind Of AGI We Want, it would have an intrinsic utilitarian preference across alternate courses of action. AIXI gives us a handwave of "Solomonoff induction" for "perfect play" in an environment, but which in practice is computationally intractable. The core problem I see is that utility functions can always be constructed post-hoc. After some revealed preferences have been spewed out by a system, you can construct a synthetic utility function with a value of 1 for that thing it did, and arbitrary-but-less-than-one values for the things it didn't do.

A lot of this is outlined in Coherent decisions imply consistent utilities. If we have a system with a coherent decision making process, its behaviours will be post-hoc describable as having consistent utility. The problem, then, is how to describe and model systems which don't have a coherent utilitarian ethics. If we can do that, and then describe necessary conditions for ones which do, we might be able to train the models which we have to approach that goal.

A utility function is the sort of thing which works when there's Exactly One Revealed Preference. In practice, LLMs output a probability distribution over next-token candidates, which we end up iteratively sampling from; in a broader sense, we can also say we're sampling across a distribution of token sequences. While in typical us...
Methods for strong human germline engineering

by TsviBT — 2025-03-03T08:13:49.414Z

PDF version. Image sizes best at berkeleygenomics.org with a wide monitor. Twitter thread

Introduction

This article summarizes the technical pathways to make healthy humans with significantly modified genomes. These are the pathways that I'm aware of and that seem plausibly feasible in the next two decades. A short summary, in a diagram:

Annotated table of contents:

Reproductive genomic vectoring explains the general idea of human germline genomic engineering, and distinguishes editing and selection.

Comparing editing and selection talks about general differences between the two kinds of genomic vectoring methods.

Reproductive GV and epigenomic correctness (EC), Methods to handle epigenomic correctness, and How GV and EC interact discuss the epigenomic correctness problem in germline engineering—what it is, why it matters, and how to address it.

Summary of genomic vectoring methods gives an annotated table of contents for the following Methods sections. The Methods sections—on Simple embryo selection, Gamete selection, Chromosome selection, Iterated recombinant selection, and Iterated multiplex CRISPR editing—give more detail about each genomic vectoring method: what it is, obstacles, variations, and how powerful it is.

The appendices give additional technical information, if you're looking around and saying "I'm not in the weeds enough, I want to be more in the weeds.".

Here's a sneak peek about the strength of different genomic vectoring methods:

The list of specific methods that this table summarizes starts after the section Summary of genomic vectoring methods. This article is roughly organized from general to specific, first discussing things that apply to the whole area, and then later discussing specific methods.

I won't lie, this article book is a bit of a slog. You try writing a book about the state of the art of realistic germline engineering in a way that is automatically fun, and then get back to me, ok? But listen: There's lots of pictures, and some of them are good pictures. You could just skim through and look at the pictures, and then read more if there's something you really want to know. If you're looking for mathy stuff, or you just want to hear about the genomic vectoring methods, start here. Also, if you have some sort of oomph you might put into this (such as brainpower or moneypower), or if you're up for a recorded and publish...
Request for Comments on AI-related Prediction Market Ideas
by PeterMcCluskey — 2025-03-02T20:52:41.114Z

I'm drafting some AI related prediction markets that I expect to put on Manifold. I'd like feedback on my first set of markets. How can I make these clearer and/or more valuable?

Question 1: Will the company that produces the first AGI prioritize corrigibility?

This question will be evaluated when this Metaculus question: When will the first general AI system be devised, tested, and publicly announced? is resolved.

At that time, I will resolve the market to YES if the organization(s) that were responsible for creating the AGI(s) that triggered the Metaculus result describe their safety approach as giving their AIs goals that put corrigibility above any other goals that the AGI might have.

This market will resolve as N/A if no AGI meeting the Metaculus criteria has been created by 2050.

I will try to evaluate this based on whether the AGI(s) were created following the spirit of Max Harm's Corrigibility As Singular Target Sequence. The AGI(s) need to be corrigible to some person or group of people, but they do not need to be corrigible to end users.

I will not trade in this market.

Question 2:
Will AGI create a consensus among experts on how to safely increase AI capabilities?

This market will resolve one year after this Metaculus question: When will the first general AI system be devised, tested, and publicly announced? is resolved.

This market will resolve as N/A if no AGI meeting the Metaculus criteria has been created by 2050.

If the Metaculus question resolves as YES, this market will resolve based on whether leading AI researchers and leading AIs say that they've agreed on a clear plan that will keep any further development of AI safe.

I plan to evaluate the safety, clarity, and extent of agreement on the plan primarily by asking three leading AIs. My planned prompt is:

Please evaluate whether at least 90% of the leading AI developers have agreed on a clear plan for ensuring the safety of any further development of AI capabilities. I plan to use this evaluation to help me resolve a Manifold Market question about when we'll get a safe approach to developing powerful AI. Try to interpret this prompt as it would have been understood in March 2025. Your answer should reflect the beliefs of both humans and AIs who have been responsible for creating the most capable AIs, as well as leaders of the most reputable AI safety organizations.
Please use an interpretation of sa

...
Statistical Challenges with Making Super IQ babies
by Jan Christian Refsgaard — 2025-03-02T20:26:22.103Z

This is a critique of How to Make Superbabies on LessWrong.

Disclaimer: I am not a geneticist[1], and I've tried to use as little jargon as possible. so I used the word mutation as a stand in for SNP (single nucleotide polymorphism, a common type of genetic variation).

Background

The Superbabies article has 3 sections, where they show:

Why: We should do this, because the effects of editing will be big
How: Explain how embryo editing could work, if academia was not mind killed (hampered by institutional constraints)
Other: like legal stuff and technical details.

Here is a quick summary of the "why" part of the original article articles arguments, the rest is not relevant to understand my critique.

we can already make (slightly) superbabies selecting embryos with "good" mutations, but this does not scale as there are diminishing returns and almost no gain past "best of 40"
They show that when you edit, you don't have the issue outlined above, because each edit helps, here the only source of diminishing returns is that you of course pick the most important mutations first.
Then they switch from IQ to a disease focus by removing bad mutations. They also show that most diseases risk can be reduced 50-100% by 3-25 mutations, but if you swap from disease association to life expectancy, then 300 edits may only gain you 14 years, Interesting it only take 16 edits to get the first 5 years.
They show that there are less tradeoffs between mutations than one might initially anticipate, the correlations are mostly positive, ie, curing one disease might protect you from another one.
Then they use chicken becoming 4 times larger over 50 year, which is 40 standard deviation! to motivate the intuition that a specie can go fare out of equilibrium, using selective breeding, so selective editing should allow us to push even further.
Finally, they make theoretical curves for IQ gain and longevity, under the assumption that if we had more data, we could more and better mutations to edit if we had stronger data.

My Position

At a high level, I appreciate the effort in the original post and think most of it is well-written. The main points about the difference between editing and selecting, the parts about diseases, and to some extent longevity, are valid. However, I am skeptical of the claimed IQ effects.

My critique can be broken down into the following subtopics:

Correlation vs. Causat

...
Cautions about LLMs in Human Cognitive Loops
by Alice Blair — 2025-03-02T19:53:10.253Z

soft prerequisite: skimming through How it feels to have your mind hacked by an AI until you get the general point. I'll try to make this post readable as a standalone, but you may get more value out of it if you read the linked post.

Thanks to Claude 3.7 Sonnet for giving feedback on a late draft of this post. All words here are my own writing. Caution was exercised in integrating Claude's suggestions, as is thematic.

Many people right now are thinking about the hard skills of AIs: their ability to do difficult math, or code, or advance AI R&D. All of these are immensely important things to think about, and indeed I spend much of my time thinking about those things, but I am here right now to talk about soft skills of AIs, so that fewer of us end up with our brains hacked by AI.

A Motivating Example

soft prerequisite for this section: Superstimuli and the Collapse of Western Civilization.

Superstimuli are stimuli that are much more intense than those in the environment where humans evolved, like how a candy bar is much denser in tasty things like sugars and fats than anything in nature.

Many humans spend much of their time following their local dopamine gradient, moving to the next most exciting thing in their immediate vicinity: they see something appealing, they go do it. They can also be strategic about things and can look to the global dopamine gradient, further away in space and time, when they need to, but this often requires nonnegligible willpower. (e.g. Stanford Marshmallow Experiment ).

Occasionally, someone gets swept up in a dopamine gradient too strong to resist, even with good reasons to stop. They overdose on drugs, they overeat unhealthy foods, they play video games for days until they die. And those are just some of the strongest dopamine gradients that humans have created.

We're seeing the beginning of the rise of cheap AI video generation. It's all over Youtube[1]. It's not good, but it's mesmerizing. It's bizarre, and it scratches some itch for some people. You can look it up if you're really morbidly curious, but I won't link anything, since the whole point of this section of the post is "don't get stuck in strong dopamine gradients from AI-generated content." When (not if) this technology does get Good, then we have cheap content generation with a powerful optimizer behind it, presumably trained well enough to grok what keeps humans engage...
Self-fulfilling misalignment data might be poisoning our AI models
by TurnTrout — 2025-03-02T19:51:14.775Z
Link: https://turntrout.com/self-fulfilling-misalignment

Your AI’s training data might make it more “evil” and more able to circumvent your security, monitoring, and control measures. Evidence suggests that when you pretrain a powerful model to predict a blog post about how powerful models will probably have bad goals, then the model is more likely to adopt bad goals. I discuss ways to test for and mitigate these potential mechanisms. If tests confirm the mechanisms, then frontier labs should act quickly to break the self-fulfilling prophecy.

Research I want to see

Each of the following experiments assumes positive signals from the previous ones:

Create a dataset and use it to measure existing models
Compare mitigations at a small scale
An industry lab running large-scale mitigations

Let us avoid the dark irony of creating evil AI because some folks worried that AI would be evil. If self-fulfilling misalignment has a strong effect, then we should act. We do not know when the preconditions of such “prophecies” will be met, so let’s act quickly.

https://turntrout.com/self-fulfilling-misalignment
Spencer Greenberg hiring a personal/professional/research remote assistant for 5-10 hours per week 
by spencerg — 2025-03-02T18:01:32.880Z

Hi all! I'm hiring for a part-time (5-10 hours per week) remote position: a professional/personal/research assistant. Please share this if you know someone that you think might be a good fit.
This is a somewhat unusual assistant role involving a wide range of projects and tasks. Some are simple, some very complex, some typical, some unconventional. This job will be an especially good fit for you if:
• you like working on a wide variety of projects
• you enjoy learning new things across multiple domains
• you're highly conscientious and productive (e.g., you're organized and get your work done efficiently and on time)
• you like having flexible work hours or like to be able to work whenever and from wherever you want
Example Work Tasks
Here are a few examples of the sort of tasks I might need your help with in this role (these are just examples; the actual work will be highly varied):
• make a list of all the meditation techniques mentioned in specific sources, and attempt to write clear step-by-step instructions (based on these sources) for how to carry out each one
• assign categories to data that we collected in one of our studies
• find an AirBnB that meets a list of criteria to book for a certain date range
• research image usage rights for physically printing images that we already have digital rights to and summarize the findings
reschedule appointments for me
• teach yourself how to use a complex piece of software you've never used before and use it to accomplish a goal
• create and then run a survey on a certain topic
• create a first draft of an essay or PowerPoint presentation based on a series of notes you took during a conversation with me
This is open to applicants all over the world - you do NOT need to live in the U.S. to apply.
For more information about the role, as well as information about how to apply, please go here:

https://docs.google.com/document/d/1VeQXhielG357dm2mSaDY75VqFyAi6vrMK7OmZbVaK6k/edit?tab=t.0
Will LLM agents become the first takeover-capable AGIs?
by Seth Herd — 2025-03-02T17:15:37.056Z

One of my takeaways from EA Global this year was that most alignment people aren't explicitly focused on LLM-based agents (LMAs)[1] as a route to takeover-capable AGI. I want to better understand this position, since I estimate this path to AGI as likely enough (maybe around 60%) to be worth specific focus and concern.

Two reasons people might not care about aligning LMAs in particular:

Thinking this route to AGI is quite possible but that aligning LLMs mostly covers aligning LLM agents
Thinking LLM-based agents are unlikely to be the first takeover-capable AGI

I'm aware of arguments/questions like Have LLMs Generated Novel Insights?, LLM Generality is a Timeline Crux, and LLMs' weakness on what Steve Byrnes calls discernment: the ability to tell their better ideas/outputs from their worse ones.[2] I'm curious if these or other ideas play a major role in your thinking.

I'm even more curious about the distribution of opinions around type 1 (aligning LLMs covers aligning LMAs) and 2 (LMAs are not a likely route to AGI) in the alignment community. [3]

Edit: Based on the comments, I think perhaps this question is too broadly stated. The better question is "what sort of LMAs do you expect to reach takeover-capable AGI?"

^

For these purposes I want to consider language model agents (LMAs) broadly. I mean any sort of system that uses models that are substantially trained on human language, similar to current GPTs trained primarily to predict human language use.

Agents based on language models could be systems with a lot or a little scaffolding (including but not limited to hard-coded prompts for different cognitive purposes), and other cognitive systems (including but not limited to dedicated one-shot memory systems and executive function/planning or metacognition systems). This is a large category of models, but they have important similarities for alignment purposes: LLMs generate their "thoughts", while other systems direct and modify those "thoughts", to both organizing and chaotic effect.

This of course includes multimodal foundation models that include natural language training as a major component; most current things we call LLMs are technically foundation models. I think language training is the most important bit. I suspect that language training is remarkably effective because human language is a high-effort distillation of the worl

...
Not-yet-falsifiable beliefs?
by Benjamin Hendricks — 2025-03-02T14:11:07.121Z

I recently encountered an unusual argument in favor of religion. To summarize:

Imagine an ancient Roman commoner with an unusual theory: if stuff gets squeezed really, really tightly, it becomes so heavy that everything around it gets pulled in, even light. They're sort-of correct---that's a layperson's description of a black hole. However, it is impossible for anyone to prove this theory correct yet. There is no technology that could look into the stars to find evidence for or against black holes---even though they're real.

The person I talked with argued that their philosophy on God was the same sort of case. There was no way to falsify the theory yet, so looking for evidence either way was futile. It would only be falsifiable after death.

I wasn't entirely sure how to respond to that, so I just indicated that this argument may be proving too much. However, it genuinely is possible for a true theory to exist that we simply lack the ability to test. Do we simply have to accept that it's impossible to learn about such things? That seems like the right answer, but it feels unsatisfying.

Any thoughts?
Saving Zest
by jefftk — 2025-03-02T12:00:41.732Z

I realized I've been eating oranges wrong for years. I cut them into
slices and eat them slice by slice. Which is fine, except that I'm
wasting the zest. Zest is tasty, versatile, compact, and freezes
well. So now, whenever I eat a navel orange I wash and zest it first:

The zest goes in a small container in the freezer, and is available
for cooking and baking as needed. Probably my favorite thing to do
with it right now is steep it in cream (~3-15g per cup, bring to near
boil, leave for 20min, filter) and then use the cream for all sorts of
things (truffles,
pastry
cream, etc). I've been meaning to try a cold infusion (24hr in
the fridge) which ought to be a bit more true to the fruit.

Comment via: facebook, lesswrong, mastodon, bluesky
Open Thread Spring 2025
by Ben Pace — 2025-03-02T02:33:16.307Z

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.