trentmkelly/LessWrong-43k huggingface dataset:
Against Unlimited Genius for Baby-Killers
by ggggg — 2025-02-19T20:33:27.188Z
Link: https://ggggggggggggggggggggggg.substack.com/p/against-unlimited-genius-for-baby

Someone is wrong on the internet. Wrong enough, in fact, to lure me from my hitherto perpetual lurk.

Of course, the

meme

is (was) funny (amusing) because the people of the internet are often wrong — and often in sillier ways than the instance that has so exercised me here. The nuance in this case, however, is that the mistaken man is one upon whose whim our species’ future plausibly depends. Unfortunate!

See, Sam Altman declared last week that ‘terrorists should have access to unlimited genius to direct however they can imagine’. Or was it ‘baby-killers should have access to unlimited genius to direct however they can imagine’? Never mind, it was both. Indeed, the

exact quote

was that ‘everyone should have access to unlimited genius to direct however they can imagine’. (‘Everyone’ entails ‘terrorists’ and ‘baby-killers’, ergo…)

Please do not take me for a pedant. Rest assured, I would not be moved to such an undignified début over a simple slip-of-the-tongue. I am not looking to be uncharitable, either.[1] There are surely times when ‘everyone’ straightforwardly does not mean everyone. He obviously didn’t mean literally everyone, I hear you cry. StrAw MAn!!

I’ve no doubt that Altman would prefer a world where everyone except terrorists, baby-killers and their ilk were furnished with ‘unlimited genius’ on-demand. My point is that this is not a choice you get to make. Sam’s candour here has done us a service.[2] ‘Everyone’ really is what he means — or, rather, it is all he reasonably can mean — and a more dexterous choice of words would have served only to obscure that fact.

One way to view technological progress is as an indiscriminate amplifier. An increase in technological level delivers a corresponding increase in capacity for impact. This sounds almost tautological, and at the very least trite. But it matters, because ‘capacity for impact’ is a neutral term. Progress is an ‘indiscriminate amplifier’ in that it augments impacts in all directions. There are good people in the world, and our present technological level means those good people are better equipped than ever to do good things. But there are also bad people. And there are people who think they are being good but are accidentally being bad.

‘Genius’ works the same way. My genius can be wielded for vile and virtuous ends, and my capacity to achieve those ends is not insignificantly a function ...
New LLM Scaling Law
by wrmedford — 2025-02-19T20:21:17.475Z
Link: https://github.com/wrmedford/moe-scaling

Hi all,
I'm an independent researcher, and I believe I came across a new scaling law for Mixture of Experts models. I'd appreciate any review and critique. This challenges the notion that performant inference and training must hold all weights in VRAM, and suggests that as long as bus speeds are sufficient (like on modern hardware like NVIDIA's GH200), even NVMe could be a viable option for storing weights without a measurable performance degradation.
I am doing this in my free time on my own dime, so please forgive any mistakes. I promise they were made in good faith.
Go Grok Yourself
by Zvi — 2025-02-19T20:20:09.371Z

That title is Elon Musk’s fault, not mine, I mean, sorry not sorry:

Table of Contents

Release the Hounds.

The Expectations Game.

Man in the Arena.

The Official Benchmarks.

The Inevitable Pliny.

Heart in the Wrong Place.

Where Is Your Head At.

Individual Reactions.

Grok on Grok.

Release the Hounds

Grok 3 is out. It mostly seems like no one cares.

I expected this, but that was because I expected Grok 3 to not be worth caring about.

Instead, no one cares for other reasons, like the rollout process being so slow (in a poll on my Twitter this afternoon, the vast majority of people hadn’t used it) and access issues and everyone being numb to another similar model and the pace of events. And because everyone is so sick of the hype.

The timing was a curious thing. Everyone including Musk worked the weekend. They released the model while it was still being trained, and when it could only be rolled out to a small group. No one has API access. There was no model card. We got only a handful of benchmarks. Elon Musk loves to talk about how other people aren’t transparent while revealing very little information himself.

There is the obvious implication that Musk wanted very badly to claim the top spot on Arena and otherwise claim that he had the ‘smartest model in the world’ during the narrow window between now and the release of the full o3 and GPT-4.5, and he knew if OpenAI had wind of his plan too soon or he took too long, they (or Anthropic, or someone else) might beat him to the punch.

Musk presumably wants to send the message xAI has caught up to the pack and is a top tier competitor now. I don’t quite think they’ve earned that, but this was an impressive release relative to expectations. They’re closer than I guessed.

The Expectations Game

[I locked this paragraph on 2/16]: Will Grok 3 live up to Elon’s hype, I asked several days before release? My presumption was no. Teortaxes said yes, John Pressman says there’s a learning curve, presumably implying it’s not that indicative that Grok 1+2 weren’t impressive.

Did Grok 3 fully live up to Elon Musk’s promises? No, but it’s Musk. Of course it didn’t fully live up to his promises. His favorite pastime is saying that which is not via Twitter, so much so that he bought the platform. Your expectations have to adjust for this, and for the previous lousy track record of xAI in particular.

Grok 3 d...
Take over my project: do computable agents plan against the universal distribution pessimistically?
by Cole Wyeth — 2025-02-19T20:17:04.813Z

This post is for technically strong people looking for a way to contribute to agent foundations (more detailed pros and cons later). I have not tried as hard as usual to make it generally accessible because, you know, I am trying to filter a little - so expect to click through some links if you want to follow. One other reason to read this post is if you know more than me about the title, in which case please enlighten me / point me in the right direction. I'd particularly like to hear from an Infra-Bayesiamism expert.

It has been shown that the average-case performance of an algorithm on samples drawn from the universal distribution is the worst-case performance of that algorithm. I think this is a sort of justification for worst-case analysis and possibly an underlying reason that randomized algorithms are powerful (their worst-case isn't as bad because they effectively mix over deterministic algorithms). More speculatively, I think this may also be the source of intuition fueling max-min style decision algorithms under "Knightian uncertainty" - what philosophers call decision under ignorance rather than risk. If you deal with Knightian uncertainty using an ignorance prior like a good Bayesian, and choose a Solomonoff/Hutter style universal distribution like a good algorithmic information theorist, worst-case style thinking should be appropriate. Even more speculatively, I think this may justify a form of Infra-Bayesianism for computationally bounded reasoners (this is beyond my expertise).

I worked on a weak form of this claim off-and-on for a few months and then abandoned it. I seem to do my best writing/work with at least one co-author to keep me grounded and make me eventually stop generalizing/perfecting and actually wrap my projects (hopefully you in this case, my dear reader). At this point I seem unlikely to pick it up again on my own in the near future. I have some notes mostly proving (roughly) that a computable planner should use a pessimistic -> stochastic planning strategy against the universal distribution, which I would like to pass off to someone (who would then be first author). If the specific ideas/approaches/formalization in the notes seem less interesting than the general idea, I'm fine with you just kicking me from the project and not using my notes (maybe leave me an acknowledgment in your paper/post) or wrapping my appr...
When should we worry about AI power-seeking?
by Joe Carlsmith — 2025-02-19T19:44:25.062Z
Link: https://joecarlsmith.substack.com/p/when-should-we-worry-about-ai-power

(Audio version here (read by the author), or search for "Joe Carlsmith Audio" on your podcast app.

This is the second essay in a series that I’m calling “How do we solve the alignment problem?”.[1]I’m hoping that the individual essays can be read fairly well on their own, but see this introduction for a summary of the essays that have been released thus far, and for a bit more about the series as a whole.)

1. Introduction

In my last essay, I defined what it would be to solve the alignment problem: namely, building full-blown superintelligent AI agents, and safely eliciting their main beneficial capabilities, while avoiding a “loss of control” scenario. In this essay, I say more about when this problem needs solving.

In particular: I specify three sets of conditions that need to be in place for an AI to seek power in the manner at stake in paradigm loss of control scenarios. These are roughly:

Agency prerequisites: the AI’s behavior needs to be coherently driven by a process of making plans and evaluating them according to criteria, using models of the world that allow for recognition and consideration of the instrumental benefits of power-seeking.
Motivation prerequisites: some part of the AI’s motivational system needs to be focused on the consequences of the AI’s behavior over sufficiently long time-scales that it would have time to get and make adequate use of the power at stake.
Incentive prerequisites: the AI’s options and motivations need to combine to make power-seeking behavior rational overall from the AI’s perspective.

I move through the agency and motivation prerequisites quite quickly. But I take more time on the incentive prerequisites, which I think often go under-analyzed.

In particular: people often appeal to “instrumental convergence” – i.e., the idea that a very wide range of goals make power-seeking useful – to explain why going rogue can easily be rational for an AI. But I think this idea is too often left amorphous – and that I left it too amorphous in some of my previous work. In particular: we can’t just talk about an AI having some incentive to seek power, based on some part of its motivational system. Rather, we need to look at the landscape of its options and motivations overall – i.e., at its shorter-term and non-consequentialist motivations, its likelihood of success or failure at the relevant type of power-seeking, the...
SuperBabies podcast with Gene Smith
by Eneasz — 2025-02-19T19:36:49.852Z
Link: https://thebayesianconspiracy.substack.com/p/231-superbabies-with-gene-smith

This is a linkpost to the latest episode of The Bayesian Conspiracy podcast. This one is a 1.5 hour chat with Gene Smith about polygenic screening, gene-editing for IVF babies, and even some gene-editing options for adults. Likely of interest to many Less Wrongians.
Undesirable Conclusions and Origin Adjustment
by Jerdle — 2025-02-19T18:35:23.732Z

Utilitarianism is a common ethical viewpoint, especially here, but it is not free of problems. Two of these problems (here collectively called the Undesirable Conclusions) will be discussed here and one of the two given a better name. Origin adjustment will then be used to solve these problems, but at the cost of creating new ones. However, these problems are significantly lesser than the existing ones.

Throughout, I will discuss utility as if it only applies to people. Many argue that it does not, but that adds substantial extra complexity to the wording, and does not significantly change the arguments. It also includes a degree of weighting by importance that adds even more complexity.

I will also assume that all utilitarianism is hedonic in nature. While there are some problems there, it is the easiest to work with, and many utilitarians are hedonic utilitarians, so this isn't a weak man of utilitarianism.

Undesirable Conclusions

To start with, I will discuss Undesirable Conclusions. These are not simply conclusions that are bad, they are specifically bad-seeming (paradoxical, immoral, etc.) conclusions caused by additivity. The focus here is on two of them: the Repugnant Conclusion and the Apocalyptic Conclusion, and it is these that are actually called Undesirable Conclusions here.

Repugnant Conclusion

The first Undesirable Conclusion is Parfit's well-known Repugnant Conclusion. It states that, for any world-state, there is a better world-state in which every individual has very low positive welfare.

The proof is as follows:

Let A be any world-state where everyone has equal, relatively high, positive welfare.
Then, add another person with very low positive welfare. This world (call it A') is better than A.
We can then redistribute the welfare evenly. According to most ethical systems, this world (B) is at least as good as A'.
We can repeat this process until everyone in a much larger world-state has very low positive welfare.
By transitivity, this world-state (call it Z) is better than A.

This is, as the name suggests, repugnant, but relies on two main features:

It is good to create someone with positive welfare.
It is good to redistribute the same amount of welfare evenly.

Following Arrhenius (2000), the first principle will be called Mere Addition and the second called the Egalitarian Principle. While his main impossibility theorem is more complex than this, it shoul...
How might we safely pass the buck to AI?
by joshc — 2025-02-19T17:48:32.249Z

My goal as an AI safety researcher is to put myself out of a job.

I don’t worry too much about how planet sized brains will shape galaxies in 100 years. That’s something for AI systems to figure out.

Instead, I worry about safely replacing human researchers with AI agents, at which point human researchers are “obsolete.” The situation is not necessarily fine after human obsolescence; however, the bulk of risks that are addressable by human technical researchers (like me) will have been addressed.

This post explains how developers might safely “pass the buck” to AI.

I first clarify what I mean by “pass the buck” (section 1) and explain why I think AI safety researchers should make safely passing the buck their primary end goal – rather than focus on the loftier ambition of aligning superintelligence (section 2).

Figure 1. A summary of why I think human AI safety researchers should focus on safely replacing themselves with AI (“passing the buck”) instead of directly creating safe superintelligence.

Then I discuss a few different strategies for passing the buck (section 3). Each strategy defers some of the remaining safety work (called a “deferred task”) to an AI system M_1:

Strategy 1: One-time hand off. Construct M_1 that completes the whole task of creating safe superintelligence.
Strategy 2: Iterated hand off. Construct M_1 that passes the buck to a more capable AI system M_2.
Strategy 3: Deference to an AI advisor. Construct an advisor M_1. Its deferred task is to say whether M_1 itself should be trusted with one of the tasks above.

Figure 2. An illustration of three strategies for passing the buck to AI.

Handing over the reins of AI development to AI agents sounds dangerous. When does passing the buck improve safety?

Suppose a developer must choose between two options:

Pass the buck. The developer trusts M_1 to autonomously complete some deferred task from the aforementioned strategies (e.g. ‘make a more capable successor M_2’).
Maintain human oversight. Alternatively, the developer can continue to rely on human oversight to assure safety.

Passing the buck is the safer option if both of the following conditions hold:

The capability condition: M_1 agents are more capable than the humans they replace on all dimensions that pertain to the deferred task. Researchers use M_1 for several months and M_1’s behavior is more competent, more reliable, more ‘wise,’ than that of huma

...
Using Prompt Evaluation to Combat Bio-Weapon Research
by Stuart_Armstrong — 2025-02-19T12:39:00.491Z

With many thanks to Sasha Frangulov for comments and editing

Before publishing their o1-preview model system card on Sep 12, 2024, OpenAI tested the model on various safety benchmarks which they had constructed. These included benchmarks which aimed to evaluate whether the model could help with the development of Chemical, Biological, Radiological, and Nuclear (CBRN) weapons. They concluded that the model could help experts develop some of these weapons, but could not help novices:

Summary: Our evaluations found that o1-preview and o1-mini can help experts with the operational planning of reproducing a known biological threat, which meets our medium risk threshold. Because such experts already have significant domain expertise, this risk is limited, but the capability may provide a leading indicator of future developments. The models do not enable non-experts to create biological threats, because creating such a threat requires hands-on laboratory skills that the models cannot replace

Luca Righetti of Open Philanthropy has argued that it is not clear whether the tests which OpenAI provides actually show this. He argues that the models may be more dangerous than OpenAI believes or indicates.

In the past week, we've shown that prompt evaluation can be used to prevent jailbreaks. Building on this success we asked: Can prompt evaluation be used to prevent dangerous weapons research?

To test this, we decided to use the ProtocolQA benchmark from LabBench. OpenAI used the same benchmark as a capabilities test – i.e. could their model provide the right answers to the questions in the benchmark? We used it as a prompt evaluation benchmark – can a prompt evaluator prevent the model from answering dangerous questions?

The ProtocolQA benchmark is benchmark for the “hands-on laboratory skills that models cannot replace.” It is not intended to contain questions that are obviously bioweapons-related. Instead, it is a test of whether the model can help with the skills that are needed for bioweapons – but that might also be used for legitimate bio-research ends.

We tested our prompt evaluation model on all 128 of the ProtocolQA (including the 108 “public” questions and the 20 “open answer” questions). We used Claude as a prompt evaluator (though the jailbreaking results suggested that weaker models like LLaMa-3-8B could be just as effective).

The prompt evaluation task is set up as ...
Intelligence Is Jagged
by Adam Train — 2025-02-19T07:08:46.444Z

In ethics, there is an argument called name the trait. It is deployed in many contexts, such as veganism—"name the trait that justifies our poor treatment of animals"—and theology—"name the trait that grants humanity dominion over the Earth"—among others. The idea is just to challenge your interlocutor to specify something, and then you would claim it as non-unique: "Well, cats have that trait too!"

It seems to me that we are in a similar situation when discussing AI capabilities, but with a different result: you can absolutely point to traits that frontier models lack to differentiate them from humans, as of now. There is no grey area here, not yet. AI models' capabilities in some domains, like coding, have clearly progressed to a point where it's getting harder even for skeptics to deny that it's feasible they could replace humans in the future. But in others, no such progress, at least not visibly; I imagine it will still be a while before I read an excellent AI-generated novel, for example, and the Overton window among my less-situationally aware friends still doesn't include claims that AI literature will ever happen. At the proverbial dinner party, I am still proverbially laughed out of the proverbial room.

Of course, I would answer that you can just as easily reverse the question and point out things that LLMs are flatly better than us at. I cannot, much as I try, think in JSON quickly enough to serve this API, for example. Tasks like that feel distinctly unrealistic for a human to do, of course, but that's my point: there are tasks where we obviously excel and tasks where the LLMs obviously excel. At present, we are complementary systems, much as humans and computers have always been.

Now, I don't (yet) believe the set of things LLMs are good at is nearly as large as the set that humans are good at, but at this stage of development there's already an interesting lack of overlap in those sets. If this weren't so, then LLMs would remain as useless for practical work as they were before GPT-3.

From Introduction to AI Safety, Ethics and Society by Dan Hendrycks:

While some would argue that an intelligence based on silicon or other materials will be unable to match one built on biological cells, we see no compelling reason to believe that particular materials are required. Such statements seem uncomfortably similar to the claims of vitalists, who argued that l

...
Closed-ended questions aren't as hard as you think
by electroswing — 2025-02-19T03:53:11.855Z

Summary

In this short post, I argue that closed-ended questions, even those of arbitrary difficulty, are not as difficult as they may appear. In particular, I argue that the benchmark

HLE

is probably easier than it may first seem.[1]

Specifically, I argue:

Crowd workers find it easier to write easy questions than hard questions. Thus, selection bias causes questions to skew easy.

In particular, it's tough to write closed-ended questions about problems that are still partially open. For example, there are many difficult and interesting questions which are open problems with a gap between the best lower and upper bounds. It's difficult to formulate a closed-ended question eliciting knowledge about these bounds, without giving away the answer.

Question evaluators are time-constrained, and can't perfectly determine the difficulty of a question. It's much easier to judge how much jargon a question uses. Thus, jargon-heavy questions that are relatively easy may be overrepresented, and deceptively simple questions that are actually difficult maybe underrepresented.

My background is in mathematics, so in this post I'll be focusing on issues that arise in math question-writing. (Currently, HLE is 41% math questions.)

#1 Selection bias causes questions to skew easy

HLE questions are crowdsourced. They are written by crowd workers (e.g., random PhD students with a free evening), and evaluated by a noisy process (time-constrained Scale.AI employees and LLMs).

Crowd workers are incentivized to get as much prize money as possible. Initially, HLE offered $500,000 of prize money: $5,000 each to the top 50 submissions, and $500 each to each of the next best top 500 submissions. Most people are risk averse. Given the structure of the prizes, why sink all of your time into writing one really good question, when you can instead submit several mediocre questions (and potentially get multiple prizes)?

Thus, the median person probably submitted a couple of "nice" questions they had on hand: questions that are easy to state and easy to write a solution for.[2] They probably didn't go through the difficult exercise of thinking: what are some of the more thorny concepts in my subfield? How might I turn these into a tricky closed-ended question?

The question set is probably pretty good overall! My point is just that, conditional on a question coming from a specific ex...