trentmkelly/LessWrong-43k huggingface dataset:
Poll on AI opinions.
by Niclas Kupper — 2025-02-23T22:39:09.027Z

TL;DR: Take polis poll here.

I made did a poll here to gather opinions on AI two years ago using pol.is. You can see my brief write up here, and a slightly updated report here (a couple more people voted after I wrote up my report).

As this last poll was two years ago the landscape and opinions thereof have changed quite a bit since. Hence, a new poll. Looking back at the old poll of two years ago it definitely seems like AGI is coming faster than a lot of people expected.

Polis tries to naturally group poll takers into groups. It found that the statement which most splits people was "I think the probability of AGI before 2040 is above 50%" (58% agreed and 30% disagreed). My guess is that now a much higher proportion would agree. I further expect that the statement which most splits people will be of a different nature. I will expand on my thoughts on the previous poll in my next write-up.

Please vote on all at least ~10-20 statements before submitting new ones. It is also worth going back to see if new statements have been made to vote on. I will hopefully write a report on the outcome in like two weeks or so. This time I will also put more effort into spreading the poll a bit more widely.
The Geometry of Linear Regression versus PCA
by criticalpoints — 2025-02-23T21:01:33.415Z
Link: https://eregis.github.io/blog/2025/02/23/geometry-pca-regression.html

In statistics, there are two common ways to "find the best linear approximation to data": linear regression and principal component analysis. However, they are quite different---having distinct assumptions, use cases, and geometric properties. I remained subtly confused about the difference between them until last year. Although what I'm about to explain is standard knowledge in statistics, and I've even found well-written blog posts on this exact subject, it still seems worthwhile to examine, in detail, how linear regression and principal component analysis differ.

The brief summary of this post is that the different lines result from the different directions in which we minimize error:

When we regress Yonto X, we minimize vertical errors relative to the line of best fit.
When we regress X onto Y, we minimize horizontal errors relative to the line of best fit.
When we plot the first principal component of X and Y, we minimize orthogonal errors relative to the line of best fit.

To understand the difference, let's consider the joint distribution of heights for father-son pairs where both are adults. When you observe the distribution of heights among adult men, you'll notice two key things.

First, height in adult men is roughly normally-distributed. While some people are taller or shorter than average, there aren't 10ft tall people or 1ft tall people. The vast majority of adult men are somewhere between 5 feet and 7 feet tall. If you were to randomly sample adult males in the US and plot the data, it would form a bell-curve shaped graph like this one:

Second, height runs in families. While there is natural variation, people with taller parents tend to be taller than average. Quantitatively, the correlation between father-son height is around 0.5 (the exact value won't matter for this post).

We can create simulated data that would resemble the actual distribution of father-son heights. We'll make the following assumptions:

Since we are only considering adult men, we'll assume the marginal distributions of fathers and sons are exactly the same. In reality, this wouldn't be quite true for various reasons (e.g., nutrition has changed over time), but it's close enough to true, and assuming this symmetry will help make certain conceptual points clearer.
Both fathers and sons are normally distributed with a mean of 69 inches (5 foot 9 in

...
Judgements: Merging Prediction & Evidence
by abramdemski — 2025-02-23T19:35:51.488Z

I recently wrote about complete feedback, an idea which I think is quite important for AI safety. However, my note was quite brief, explaining the idea only to my closest research-friends. This post aims to bridge one of the inferential gaps to that idea. I also expect that the perspective-shift described here has some value on its own.

In classical Bayesianism, prediction and evidence are two different sorts of things. A prediction is a probability (or, more generally, a probability distribution); evidence is an observation (or set of observations). These two things have different type signatures. They also fall on opposite sides of the agent-environment division: we think of predictions as supplied by agents, and evidence as supplied by environments.

In Radical Probabilism, this division is not so strict. We can think of evidence in the classical-bayesian way, where some proposition is observed and its probability jumps to 100%. In the betting-market analogy, this is like the judge resolving a betting contract. However, Radical Probabilism encourages us to think of "softer" evidence, which pushes our beliefs some distance in a direction but not to 100% or 0%.

We can think of a judge in a betting market as just another market participant, who happens to have arbitrarily much money. Resolving a bet is like offering to buy out the contract for its payout value. If we think of the opposite side of the bet as a short, the value of the short goes to zero, because the judge will tirelessly buy up the other side of that short.[1]

This means we can think of some market participants as hypothesis-like, and others as evidence-like. The hypothesis-like market participants are profit-seekers who attempt to use their own wits to predict future prices. The evidence-like market participants are not profit-motivated; instead, they spend their money on ensuring that the market prices reflect the truth. They perceive external events and buy up the corresponding market goods; when they see X, they buy shares in X.

However, the main point here is not to make the distinction, but rather, to point out that it is fuzzy and not absolute. I propose the term judgement to unify prediction & evidence, because "judging a judgement" makes sense (IE, judgements can intuitively play either the prediction role or the evidence role or both).

If everything so far made perfect sense, you ...
Intelligence as Privilege Escalation
by Cole Wyeth — 2025-02-23T19:31:27.604Z

Epistemic status: An interesting idea that is probably already in the air.

Inherent power you possess as part of yourself. Granted power is lent or given by other people.

-Patrick Rothfuss, The Wise Man's Fear

Humans are more powerful than other animals because we are smarter - and better coordinated. Both sides of the story are important for understanding human power on an individual level as well.

Intelligence is a powerful thing. It has allowed humans to collectively reshape our world by inventing technologies that outstrip pretty much every animal at their own specialization. Rationalists have a pretty strong norm of optimizing intelligence over physical attributes like strength, exemplified and often justified by Eliezer's argument that humans conquered the world by being smarter, not by having sharper claws than other animals. Certainly on the scale of conflict between other species and humanity this holds up, and I think it works on the individual scale as well. That is, if you want the power to personally move really fast, you could practice sprinting or you could go work for a hedge fund and buy a sports car.

Notice the difference in frame here - I am focused on what makes an individual powerful in our context: the Anthropocene, a time when the world has been drastically reshaped and reorganized to serve human needs, with multiple levels of organization built on top of the material "resource extraction" layer, from markets to governments. This is different from what makes our species powerful; the power of our species is only the backdrop and the playing field.

Now, it is standard rationalist doctrine that things like being "cool" or charismatic ultimately happen in the brain, and are therefore facets of intelligence. I think this is mostly right; a decent percentage of the social skills that might be described as "charm" are primarily a type of social intelligence. It would be a mistake to argue that because a lot of anti-social unwashed nerds are less influential than famous youtubers with shining smiles and basic but entertaining takes, intelligence doesn't get you anywhere or is even detrimental. In fact, the youtubers are engaged in a mental struggle of their own (against eachother and the Youtube algorithm). Their success is also based partially on intelligence.

But I think this picture of intelligence as the primary driver of individual power[...
Have LLMs Generated Novel Insights?
by abramdemski — 2025-02-23T18:22:12.763Z

In a recent post, Cole Wyeth makes a bold claim:

. . . there is one crucial test (yes this is a crux) that LLMs have not passed. They have never done anything important.

They haven't proven any theorems that anyone cares about. They haven't written anything that anyone will want to read in ten years (or even one year). Despite apparently memorizing more information than any human could ever dream of, they have made precisely zero novel connections or insights in any area of science[3].

I commented:

An anecdote I heard through the grapevine: some chemist was trying to synthesize some chemical. He couldn't get some step to work, and tried for a while to find solutions on the internet. He eventually asked an LLM. The LLM gave a very plausible causal story about what was going wrong and suggested a modified setup which, in fact, fixed the problem. The idea seemed so hum-drum that the chemist thought, surely, the idea was actually out there in the world and the LLM had scraped it from the internet. However, the chemist continued searching and, even with the details in hand, could not find anyone talking about this anywhere. Weak conclusion: the LLM actually came up with this idea due to correctly learning a good-enough causal model generalizing not-very-closely-related chemistry ideas in its training set.
Weak conclusion: there are more than precisely zero novel scientific insights in LLMs.

My question is: can anyone confirm the above rumor, or cite any other positive examples of LLMs generating insights which help with a scientific or mathematical project, with those insights not being available anywhere else (ie seemingly absent from the training data)?

Cole Wyeth predicts "no"; though LLMs are able to solve problems which they have not seen by standard methods, they are not capable of performing novel research. I (Abram Demski) find it plausible (but not certain) that the answer is "yes". This touches on AI timeline questions.

I find it plausible that LLMs can generate such insights, because I think the predictive ground layer of LLMs contains a significant "world-model" triangulated from diffuse information. This "world-model" can contain some insights not present in the training data. I think this paper has some evidence for such a conclusion:

In one experiment we finetune an LLM on a corpus consisting only of distances between an unknown city and other k

...
The case for corporal punishment
by Yair Halberstadt — 2025-02-23T15:05:28.149Z

Preceded By: The case for the death penalty

Scott's essay https://www.astralcodexten.com/p/prison-and-crime-much-more-than-you is going to be my main source for most of my claims here, I recommend reading it either before or after.

Prison is very expensive, on the order of $100,000 per person per year. So what are we getting for our money?

I know of 4 main purposes of prison[1].

Deterrence: The threat of prison may discourage criminals from committing crimes.
Incapacitation: Whilst a criminal is in prison they cannot commit (most) crimes.
Aftereffects: Prison may reform criminals into the sort of person who doesn't commit crimes.
Policing: Knowing the criminals they catch get put in prison encourages police to do their jobs. Police presence is actually one of the most effective ways of preventing crime, but there's got to be something backing them up.

Unfortunately it's not really cost effective for any of them.

The deterrence effect of prison sentences appears to be weak.
Prison very effectively incapacitates criminals but the cost is linear in the amount of time you incapacitate them. Further, the chance a criminal commits a crime once they leave prison does not drop the longer they're in prison, so if incapacitation is worth the costs, then it's worth it for life[2]. There's no reason (from an incapacitation perspective) to let someone go after 5 years. Putting someone in prison for life costs millions of dollars.
If anything, prison increases the chance criminals commit further crimes. This is likely because they've lost their jobs, homes and families, but gained lots of criminal friends.
Since prisons are expensive and overflowing, most petty criminals are let off with a slap on the wrist. Whilst doing so is cost effective deterrence, this discourages police, giving them the impression that petty crimes are not worth following up on.

In those situations where it's worth paying extremely high costs to prevent someone even being able to commit a crime, my preferred solution is to execute them.

But how do we cheaply and effectively deter thieves, in a way that shows police their work is effective, and doesn't throw a wrench in criminal's lives that might force them down the wrong path?

The answer is obvious. Instead of locking a criminal up, thereby causing them a small rate of pain/unpleasantness over a long period of time, find something that causes a similar total amo...
Reflections on the state of the race to superintelligence, February 2025
by Mitchell_Porter — 2025-02-23T13:58:07.663Z

My model of the situation is that some time last year, the frontier paradigm moved from "scaling up large language models" to "scaling up chain-of-thought models". People are still inventing new architectures, e.g. Google's Titans, or Lecun's energy-based models. But it's conceivable that inference scaling really is the final paradigm prior to superintelligence. If it can produce von Neumann-level intelligence, then that should be the end, right? - the end, in the sense that control will pass out of human hands, unless there are a few humans embedded in the self-transformations of these genius-level AIs, and those embedded humans will become something more than human or other than human quickly enough.

This leads to the concrete question, where are the new frontier models, the powerful chain-of-thought AIs, being produced? Because these organizations are also the contenders to produce the first superintelligence. My list of known or suspected organizations consists of 5 in the USA, 1 in China, and 1 in Israel. Of course there may be more.

At the head of my list is Elon Musk's xAI. It's at the head of the list, not because I believe that Grok 3 is "the world's smartest AI", but because of its political advantages in Trump 2.0 America. xAI is part of the Musk family of companies, and these are now more deeply integrated into US government activities than ever before. There is a serious possibility that superintelligence will be "born" commanding all the resources of Musk's unprecedented empire, including social media, rocketry, robotics, brain-computer interfaces, and US government information systems.

Intellectually, Trump 2.0 means the ascendancy to power of a large number of ideas from outside the institutional consensus of the liberal establishment (which includes at a minimum, the universities, the mainstream media, and all parts of the government): peace with Putin's Russia, RFK Jr's idea of health, downsizing the government, reversal of DEI policies, and probably more to come. In the relatively esoteric area of frontier AI policy, it is as if e/acc replaced effective altruism as the implicit zeitgeist. Concretely that seems to mean switching from regulation (of AI) and international treaties, to deregulation and a competitive race with China.

Again xAI has a bit of an advantage here, because Musk made X-Twitter friendly to these outs...
List of most interesting ideas I encountered in my life, ranked
by Lucien — 2025-02-23T12:36:48.158Z

Bayesian thinking
It litterally was n°1 in my list, I was really happy to find this website, will not detail.
Active ignorance/avoidance and selective attention/participation
Instead of thinking, commenting, or saying something is stupid/bad, ignore/block it and just talk about the other thing that is better. Because just by thinking about/mentionning the thing, you propagate it, starting in your memory and to a greater extent in the world. This is applicable to ideas, people, activities, etc.
Participating in something is accepting that that thing is worth participating in.
Paying (money, time, attention, action) is voting/supporting.
Source Criticism
When reading information, take into account who created that information and what are their environment/interests/incentives.
Memetic Evolution
Natural selection and evolution applied to ideas.
Subjectivity of Value
Value of a thing is subjective. Loopy example is money: everyone wants money because everyone wants money.
Life as negentropy
The definition of life is to locally reduce entropy/slow entropy/maximise possibilites while minimising energy consumption/will-to-power in Nietzsche's words. And maybe reversing entropy.
Identity as Action, Not Essence
Human identity is a consequence of doing/being, not doing/being is a consequence of identity. What you do defines who you are, not who you are defines what you do.
When you die, people come at your burial and the next day, they go back to work.
Emotions are a deprecated way of making decision. Not intuition.

History of the entire world, I guess
Curious about other people's list.
Test of the Bene Gesserit
by lsusr — 2025-02-23T11:51:10.279Z

Jessica didn't say what she felt, only what her son needed to hear.

"Paul…," Jessica said it with love, "You are going to die. But remember you are a duke's son. Do not dishonor Leto with your passing." Jessica whirled and strode from the room with a swish of her skirt. The door closed with a satisfying thunk, leaving the boy alone with the crone.

Each waited for the other to speak. It was a power game older than human beings, older than primates.

"Now that we are alone, shall we drop our masks?" asked Paul.

Instead of speaking, the Reverend Mother reached into the folds of her gown and extracted a tarnished copper cube 16 centimeters in diameter. One side appeared to be missing. Inside was just black, as if a square portal had been cut into a parallel plane of infinite emptiness.

"Put your right hand into the box," the Reverend Mother said.

Paul willingly chose to place his hand into the box. Of course, the Reverend Mother knew he would do that. So was it really a choice? Paul felt a soft caress close around his hand, then a light prickling sensation. Finally nothing, as if his hand had been cleanly amputated.

The woman smiled cruelly. She raised her right hand to his neck. Paul instinctually flinched away.

"Stop," the woman commanded, using the Voice. It trumped Paul's reflex. Then he felt the tip of a needle painfully prick his skin. Paul was glad his combat trainer Gurney Halleck wasn't here to see this. Nevermind that Paul's mother has set him up.

Paul smiled. "Now are the conditions adequate for us to finally be honest with one another?" Paul's other mentor, Thufir Hawat, always told Paul that the best way to persuade a person to truth was to stick out your throat. Hawat had been speaking metaphorically, but nevermind that.

"Do not worry," said the Reverend Mother, "We of the Bene Gesserit never kill anyone of value."

"Do I have value?" asked Paul. He knew the answer. He was just dancing with words. They both knew that.

"We shall find out," said the Reverend Mother, "Now, be silent."

If this were one of Hawat's games, then Paul knew that the proper answer would be say something—anything. After all, a Rationalist should never be silent just because he was told to be. But today was not a Rationalist game, it was a Bene Gesserit exam. Still, the unconscious habit to protest was there. Paul stuffed it down.

"Good," said the Reverend Mother, "You pass the first test. You...
Moral gauge theory: A speculative suggestion for AI alignment
by James Diacoumis — 2025-02-23T11:42:31.083Z

[Epistemic status: Speculative.

I've written this post mostly to clarify and distill my own thoughts and have posted it in an effort to say more wrong things.]

Introduction

The goal of this post is to discuss a theoretical strategy for AI alignment, particularly in the context of the sharp left-turn phenomenon - the idea that AI systems will be aligned on in-distribution data but risk misalignment when extended to novel regimes. Current alignment strategies, such as Reinforcement Learning from Human Feedback (RLHF), attempt to mitigate this by averaging over human evaluations to create reward functions. However, these approaches are fundamentally limited - they rely on messy, subjective human judgments and fail to address the deeper issue of generalisation failures. In this post, I propose that by leveraging concepts from physics - specifically, the invariance and conservation laws emerging from gauge symmetries - we might be able to design reward functions that are inherently robust against such generalisation failures.

Motivation: Messily specified reward functions

The RLHF paradigm in AI alignment specifies a reward function R(x,y) where x is some input text, y is some output text and the function R gives a scalar which denotes how well the model's output y matches the given input x.

This function is brittle for several reasons:

Human subjectivity: What humans rate as "good" may be based on their mood, cultural background or any number of specific biases.
Moral disagreement: Even if anchored to an objective moral framework, moral theories disagree on many meta-ethical issues.
Generalisation failures: A reward function may not be correctly extrapolated when AI intelligence generalises out of distribution.
Goodhart's Law: A proxy reward may be misaligned with the true goal, particularly when the proxy diverges from the true goal out of distribution.

Current RLHF implementations attempt to solve (1) and (2) by averaging over large datasets of human evaluations in the hope that a large enough sample-size and an "averaging" effect produces a coherent middle-ground. However, this approach is insufficient for solving (3) i.e. ensuring that the reward function generalises correctly as AI intelligence scales out of distribution. Averaging over different moral frameworks cannot guarantee alignment through sharp left turns...