trentmkelly/LessWrong-43k huggingface dataset:
Numberwang: LLMs Doing Autonomous Research, and a Call for Input
by eggsyntax — 2025-01-16T17:20:37.552Z

Summary

Can LLMs science? The answer to this question can tell us important things about timelines to AGI. In this small pilot experiment, we test frontier LLMs on their ability to perform a minimal version of scientific research, where they must discover a hidden rule about lists of integers by iteratively generating and testing hypotheses. Results are ambiguous: they're mostly pretty bad at it but top systems show apparent signs of life. We're working on a larger, more rigorous experiment, and we really want your input.

Structure

In this post we:

Describe an experiment on general reasoning and scientific research ability in LLMs.
Describe the main research project for which this is a pilot project.
Ask for your predictions on the outcome of the main project and what you believe it will say about general reasoning in LLMs.

This is followed by appendices with more detail (eg related work, limitations) but we've kept the main body as short and direct as possible.

Introduction

Over the past six months we have been trying to better understand the degree to which LLMs are capable of general reasoning[1] (in LLM Generality is a Timeline Crux, and LLMs Look Increasingly Like General Reasoners). In short: researchers, including AI safety researchers, have widely differing positions on whether (and to what degree) LLMs are capable of the same sort of accurate and broadly applicable reasoning ability that humans are. At one extreme is the stochastic parrot hypothesis that "no actual language understanding is taking place in [LLMs]" and any apparent signs of reasoning are actually a kind of cheating. At the other extreme is the view that LLMs are already fully capable of all the same types of reasoning as humans are and just need a bit more scaling.

An important negative consequence of this disagreement is that the AI safety community's limited resources are spread across a range of timelines, in a way that impacts the scaling labs much less. The best use of those resources is significantly different if LLMs will scale straight to AGI[2], relative to the worlds where substantial further breakthroughs are needed first[3].

In order to try to improve our understanding of this issue and help resolve the disagreement, we created the Numberwang[4] experiment.

The experiment

This is a pilot experiment for a larger research project which started this month and runs through...
Topological Debate Framework
by lunatic_at_large — 2025-01-16T17:19:25.816Z

I would like to thank Professor Vincent Conitzer, Caspar Oesterheld, Bernardo Subercaseaux, Matan Shtepel, and Robert Trosten for many excellent conversations and insights. All mistakes are my own.

I think that there's a fundamental connection between AI Safety via Debate and Guaranteed Safe AI via topology. After thinking about AI Safety via Debate for nearly two years, this perspective suddenly made everything click into place for me. All you need is a directed set of halting Turing Machines!

Motivating Example

... Okay, what?? Let's warm up with the following example:

Let's say that you're working on a new airplane and someone hands you a potential design. The wings look flimsy to you and you're concerned that they might snap off in flight. You want to know whether the wings will hold up before you spend money building a prototype. You have access to some 3D mechanical modeling software that you trust. This software can simulate the whole airplane at any positive resolution, whether it be 1 meter or 1 centimeter or 1 nanometer.

Ideally you would like to run the simulation at a resolution of 0 meters. Unfortunately that's not possible. What can you do instead? Well, you can note that all sufficiently small resolutions should result in the same conclusion. If they didn't then the whole idea of the simulations approximating reality would break down. You declare that if all sufficiently small resolutions show the wings snapping then the real wings will snap and if all sufficiently small resolutions show the wings to be safe then the real wings will be safe.

How small is "sufficiently small?" A priori you don't know. You could pick a size that feels sufficient, run a few tests to make sure the answer seems reasonable, and be done. Alternatively, you could use the two computationally unbounded AI agents with known utility functions that you have access to.

Let's use the two computationally unbounded AI agents with known utility functions. One of these agents has the utility function "convince people that the wings are safe" and the other has the utility function "convince people that the wings will snap." You go to these agents and say "hey, please tell me a resolution small enough that the simulation's answer doesn't change if you make it smaller." The two agents obligingly give you two sizes.

What do you do now? You pick the smaller of the two! Whichever ...
AI #99: Farewell to Biden
by Zvi — 2025-01-16T14:20:05.768Z

The fun, as it were, is presumably about to begin.

And the break was fun while it lasted.

Biden went out with an AI bang. His farewell address warns of a ‘Tech-Industrial Complex’ and calls AI the most important technology of all time. And there was not one but two AI-related everything bagel concrete actions proposed – I say proposed because Trump could undo or modify either or both of them.

One attempts to build three or more ‘frontier AI model data centers’ on federal land, with timelines and plans I can only summarize with ‘good luck with that.’ The other move was new diffusion regulations on who can have what AI chips, an attempt to actually stop China from accessing the compute it needs. We shall see what happens.

Table of Contents

Table of Contents.

Language Models Offer Mundane Utility. Prompt o1, supercharge education.

Language Models Don’t Offer Mundane Utility. Why do email inboxes still suck?

What AI Skepticism Often Looks Like. Look at all it previously only sort of did.

A Very Expensive Chatbot. Making it anatomically incorrect is going to cost you.

Deepfaketown and Botpocalypse Soon. Keep assassination agents underfunded.

Fun With Image Generation. Audio generations continue not to impress.

They Took Our Jobs. You can feed all this through o1 pro yourself, shall we say.

The Blame Game. No, it is not ChatGPT’s fault that guy blew up a cybertruck.

Copyright Confrontation. Yes, Meta and everyone else train on copyrighted data.

The Six Million Dollar Model. More thoughts on how they did it.

Get Involved. SSF, Anthropic and Lightcone Infrastructure.

Introducing. ChatGPT can now schedule tasks for you. Yay? And several more.

In Other AI News. OpenAI hiring to build robots.

Quiet Speculations. A lot of people at top labs do keep predicting imminent ASI.

Man With a Plan. PM Kier Starmer takes all 50 Matt Clifford recommendations.

Our Price Cheap. Personal use of AI has no meaningful environmental impact.

The Quest for Sane Regulations. Weiner reloads, Amodei genuflects.

Super Duper Export Controls. Biden proposes export controls with complex teeth.

Everything Bagel Data Centers. I’m sure this ‘NEPA’ thing won’t be a big issue.

d/acc Round 2. Vitalik Buterin reflects on a year of d/acc.

The Week in Audio. Zuckerberg on Rogan, and several sound bites.

Rhetorical Innovation. Ultimately we are all on the same side.

Aligning a Smarter

...
Deceptive Alignment and Homuncularity
by Oliver Sourbut — 2025-01-16T13:55:19.161Z

NB this dialogue occurred at the very end of 2023, and for various reasons is only being published ~a year later! Keep this in mind while reading.

Key topics covered

consistent NN-internal deceptive planning (search: goblin, fae, homunc, inner)
making AI/agents from foundation models (search: scaffold, condition)
planning (search: planning, simulator, consequential)
standards of evidence (search: fanfic, theory, Shrek, analog, logic)
elicitation gaps (search: censor, creativity)

Oliver Sourbut

Background

Alex wrote a LW shortform about (flimsiness of) arguments for deceptive alignment. Someone snapshotted a bit on Twitter.

Deceptive alignment seems to only be supported by flimsy arguments. I recently realized that I don't have good reason to believe that continuing to scale up LLMs will lead to inner consequentialist cognition to pursue a goal which is roughly consistent across situations. That is: a model which not only does what you ask it to do (including coming up with agentic plans), but also thinks about how to make more paperclips even while you're just asking about math homework.

Aside: This was kinda a "holy shit" moment, and I'll try to do it justice here. I encourage the reader to do a serious dependency check on their beliefs. What do you think you know about deceptive alignment being plausible, and why do you think you know it? Where did your beliefs truly come from, and do those observations truly provide

P(observations∣deceptive alignment is how AI works)P(observations∣deceptive alignment is not how AI works)>>1?

I tweeted a response.

I (and I think others) take this to be a non-sequitur. LLM alone isn't the problem. Conditioned LLM (which is ex hypothesi goal-pursuing), or scaffolded LLM, or some yet-to-be-designed goal-pursuing system is the problem. (But externalised reasoning might help oversight...? yay!)

Conditioning (whether by prompting, fine-tuning of one or another kind, sampling, search, scaffolding, ...) is just part of the process of development in LLM-land. We see that appropriately-conditioned LLMs can be deceptive (ARC evals). So yes, clearly likelihood ratio >> 1 ?

TurnTrout

Someone showed me some of your recent tweets on deceptive alignment stuff. I try to minimize twitter in order to, you know, have a good life. I wanted to clarify: I agree that if you finetune a model to deceive you, that gets you deception. I'm n

...
Introducing the WeirdML Benchmark
by Håvard Tveit Ihle — 2025-01-16T11:38:17.056Z

WeirdML website

Related posts:

How good are LLMs at doing ML on an unknown dataset?

o1-preview is pretty good at doing ML on an unknown dataset

Introduction

How good are Large Language Models (LLMs) at doing machine learning on novel datasets? The WeirdML benchmark presents LLMs with weird and unusual machine learning tasks, designed to require careful thinking and actual understanding to solve, and tests an LLM's ability to:

Actually understand the properties of the data and the problem
Come up with an appropriate ML architecture and training setup for the problem, and generate working PyTorch code that implements the solution
Debug and improve the solution over 5 iterations based on terminal output and the accuracy on the test set
Make good use of limited computational resources and time

Each task comes with a task prompt describing the problem precisely and some example code for loading data and saving predictions. The different tasks pose various challenges: some require heavy data augmentation, others need careful feature engineering, or require combining information from many different parts of the input.

Results

Average accuracy across all tasks for each model. Grey markers indicate performance on individual tasks, bars show the mean across tasks.

Average accuracy across all six tasks for each model.

Evaluation Setup

The evaluation uses an automated pipeline that:

Presents the task to the LLM
Executes the generated code in an isolated environment
Evaluates the results against the test set

Provides feedback (terminal output from the code execution and test accuracy) to the LLM for improvement

Evaluation pipeline showing the flow from LLM code generation through isolated execution to metric evaluation and feedback, with fixed computational constraints enforced via Docker

System Architecture

The system executes code in a Docker container with strict resource limits (TITAN V GPU with 12GB memory, 600-second timeout). This ensures fair comparison between models and tests their ability to work within realistic constraints.

Each 'run' is 5 iterations, i.e., the LLM gets 5 submissions, and 4 rounds of feedback, allowing them to learn from feedback and improve their solutions (full system prompt). The accuracy of each run is the maximum test accuracy achieved over all the 5 submissions in that run.

For each task we give each model (at least) 15 runs (due to the high cost, o1-pr...
The Mathematical Reason You should have 9 Kids
by Zero Contradictions — 2025-01-16T11:24:46.617Z
Link: https://eternalanglo.com/have-nine-kids

In this post I propose a curious genetic question that can be modeled with a remarkably simple answer. If you have children, what is the probability that every allele in your genome is present in at least one of your children? In other words, if you have children, what is the probability that your entire genome has been replicated in the next generation?

Note: I do not believe there is a correct number of children to have.

This blog post is just for fun

. An organism's biological purpose is not to replicate its genome. Rather, an organism's biological purpose is simply to reproduce.
For an explanation of biological purpose, I invite you to read Debunking the Selfish Gene by T. K. Van Allen.

As a human, you have two sex chromosomes XY or XX, and you have 22 homologous pairs of autosomal chromosomes (autosomes) numbered 1 through 22. Your child receives 23 chromosomes from you and 23 chromosomes from the other parent. The manner in which each chromosome is transmitted to your child is independent of the manner in which the other chromosomes are transmitted to your child. Each autosome has a roughly probability of being transmitted as a crossover of your own homologous autosome pair during cellular meiosis. And each autosome has a roughly probability of being transmitted as an identical copy of 1 of your 2 corresponding homologs during cellular meiosis.

In this image, chromosomes of different size correspond to differently numbered chromosomes. Chromosomes of the same size and different single colors correspond to homologs of the same numbered chromosome. The H-shaped things are two chromatids attached at a centromere, and the 1-shaped things are lone chromatids. Dual-colored chromatids were generated by a crossover event during meiosis I. The end result of meiosis is 4 gamete cells. A single gamete cell from each of 2 parents fuse to form the zygote.
Source: Wikimedia Commons

Because chromosomes are transmitted independently, the probability that all of your autosome pairs are replicated into your children is just the probability that one of your autosome pairs is replicated into your children, raised to the power of 22.

(see the rest of the post in the link)
Replicators, Gods and Buddhist Cosmology
by KristianRonn — 2025-01-16T10:51:14.268Z

From the earliest days of evolutionary thinking, we’ve used metaphors to understand how life changes over time. One of the most enduring is the image of a vast “fitness landscape” with countless peaks and valleys, each corresponding to different levels of survival and reproduction. This landscape is a way of imagining how living things (the “replicators”) search for better solutions to life’s challenges—through trial, error, and the mindless mechanics of mutation and selection.

At the heart of it all are replicators: entities that copy themselves (with variation) and compete for whatever resources they need to persist. In the primordial soup, they might have been self-copying molecules jockeying for nutrients, while in modern ecosystems, they range from viruses hijacking cells to human beings shaping civilizations. Despite all this diversity, their basic “goal,” if we can call it that, is constant: remain in the game and keep making copies.

But here’s the interesting part: evolution doesn’t get to see the entire map. It is a blind watchmaker. It doesn’t know where the next mountain peak might be, nor can it foresee an impending drop into a valley. All it does is take one step at a time, test the new terrain, and keep the changes that happen to work better than whatever came before. In effect, each species’ journey across the fitness landscape looks like a

random walk

—a blind, incremental exploration of what might lead to more offspring.

Evolution through natural selection operates on populations of individuals. When we zoom in on the smallest steps of this process, we observe individual organisms and the random genetic mutations that arise each generation. While each mutation is ultimately determined by the laws of physics, its impact on overall fitness can feel as random as a roll of the dice. Some are neutral, some harmful, and occasionally a few are beneficial, granting a slight edge in survival and reproduction.

Imagine a group of individuals each taking a small step in a different direction on the landscape. Most of these new paths will lead nowhere special—many might wander straight into a valley of lower fitness, where individuals eventually die off or fail to leave many offspring. But from time to time, a single step hits higher ground, granting an advantage that can spread through the population over generations. Although many i...
Quantum without complication
by Optimization Process — 2025-01-16T08:53:11.347Z

Learning quantum mechanics involves two things:

learning the fundamentals, the nature of the stuff of which Nature is made; and
loads of universe-imposed incidental complexity, spinors and vector calculus and Lie algebras and Hilbert spaces, in order to be able to effectively apply the fundamentals to the real world.

Now, me, personally, I really like the feeling of understanding how stuff fundamentally works, even if I'm missing the accumulated genius-lifetimes' worth of tricks-I-could-theoretically-work-out-from-first-principles that I'd need in order to make that fundamental understanding practically useful. I loved Introduction to abstract entropy and Generalized Heat Engine.

This post is an attempt to treat quantum mechanics in a similar way: exploring it with mathematical precision in an aggressively-stripped-down universe without all the complications ours has. It might feel like tooling around pointlessly in abstract-theory-land (I enjoy doing that sometimes), but our toy models will produce some behavior that sure looks a lot like some classic real-world quantum stuff.

(Target audience: handy with vectors. Ideally knows what an eigenvector is, if not which end to hold. Has a rough understanding of quantumy stuff, like, maybe took a physics class which spent a week or two on it.)

(Epistemic status: was a physics major in college, did quite well, but am rusty. Take my money if you think I've made a mistake!)

Basics: the classical universe

Here's a simple system, almost the simplest thing worthy of being called a "system":

It has 6 states it can be in: A through F. The arrows depict the dynamics of the system: on each time-step, the system transitions to whatever state its current state's arrow points to.

Some example paths the system might take through time, depending on which state it starts in:

A→B→C→A→B→C→A→⋯
D→B→C→A→⋯
F→F→F→⋯
E→F→F→⋯

Here are some more example systems, just to give a sense of what's possible:

(Left to right, top to bottom: a system with one state which it's stuck in; a system that flip-flops between its two possible states; doodling; doodling; a system with an infinite number of states, 'cause why not; and another one with infinite states, just with a more interesting structure.)

(Aside: tying this back to our universe: if we were to use a diagram like this to depict the physics of, say, a ball bouncing around in a box, ...
Permanents: much more than you wanted to know
by Dmitry Vaintrob — 2025-01-16T08:04:00.510Z

Today's "nanowrimo" post is a fun longform introduction to

permanents

and their properties, organized in the way I wish it had been explained to me. Note before going on: this is not my field, and this post is even more likely than usual to contain incorrect statements or bugs.

Recall that given an n×n matrix A with coefficients Aij, its determinant is a sum over permutations σ on n letters with j=σ(i), det(A)=∑σ(−1)σ∏iAiσ(i). Here the notation (−1)σ denotes the

parity

of a permutation. Determinants are ubiquitous and extremely useful. They are also computable in polynomial time.

The permanent is defined the same way as the determinant, but without the signs (−1)σ: perm(A)=∑σ∏iAiσ(i). It is neither ubiquitous nor computable in polynomial time.

When I first heard of it in the pure math community, it was described to me as the ugly stepsister of the determinant: an object with “no elegant meaning”: the result of an idle question of “whoa, what if no signs”. The most famous result about the permanent is

Leslie Valiant’s proof

that it is NP hard (and more specifically, #P complete; #P is a more sophisticated version of NP that we’ll discuss later). I knew about this result, but thought of it as one of the dime-a-dozen theorems that show that inelegant problems lie in inelegant complexity classes.

I’ve since updated. As I’ll hopefully get across, permanents are actually quite elegant artifacts if thought about correctly, namely in the context of multiparameter polynomial algebra. I’ll explain how to think about their properties in this context, and what interesting things they say about high-dimensional polynomial algebra in general.

But first as a teaser, I’ll explain the #P class and why Valiant’s proof, far from being a routine “complexity check” of an arbitrary problem should be thought of as a pretty mind-blowing insight.

#P and Santa’s factory

In order to introduce the #P complexity class, let me tell you a story about Santa and his elves. I promise this story will be relevant. Namely, as is widely known, there is a singularity at the North pole of the Earth. At this singularity lies Santa’s workshop, which produces exponential quantities of content for the exponential quantum branches of boys and girls in the multiverse. In order to meet demand, Santa keeps a factory of exponentially many elves. The elves are very well-treat...
Gaming TruthfulQA: Simple Heuristics Exposed Dataset Weaknesses
by TurnTrout — 2025-01-16T02:14:35.098Z
Link: https://turntrout.com/original-truthfulqa-weaknesses

(Explanation. Also I have no reason to think they hate me.)

Do not use the original TruthfulQA multiple-choice or the HaluEval benchmark. We show that a simple decision tree can theoretically game multiple-choice TruthfulQA to 79.6% accuracy—even while hiding the question being asked! In response, the TruthfulQA authors created a new multiple-choice condition which avoids the vulnerabilities we highlight.

https://turntrout.com/original-truthfulqa-weaknesses