trentmkelly/LessWrong-43k huggingface dataset:
Problems with Bayesianism: A Socratic Dialogue
by B Jacobs — 2025-04-22T14:09:57.215Z
Link: https://bobjacobs.substack.com/p/solutions-to-problems-with-bayesianism

Crossposted from my blog

In this fictional dialogue between a Bayesian (B) and a Non-Bayesian (N) I will propose solutions to some pre-existing problems with Bayesian epistemology, as well as introduce a new problem for which I offer a solution at the end. (Computer scientists may consider skipping to that section).

Here’s a Bayes theorem cheat sheet if you need it:

The problem of confirmation

Nayesian: Remind me again, how does confirmation work in Bayesianism?

Bayesian: The evidence E confirms hypothesis H if and only if the posterior probability of H given E is greater than the prior probability of H.

N: So there is no difference between increasing the hypothesis’ probability and confirming a hypothesis?

B: Not really, no.

N: I would say that there are cases where something increases the probability of a hypothesis, but we would not say that it confirms the hypothesis. Let's say there is a recent hypothesis that has a strong theoretical foundation and some evidence from a few experiments. Scientists disagree about whether or not this hypothesis is correct. But, again, it's a novel proposal. Say that a recent article published in a highly regarded scientific journal supports this hypothesis. Appearing in the journal increases my degree of belief that the hypothesis is true. So it appears that the Bayesian must conclude that a publication in a reputable scientific journal confirms the hypothesis, but surely that is incorrect. Appearing in a reputable scientific journal isn't in and of itself evidence that a hypothesis is correct; it merely implies that there exists evidence supporting the hypothesis.

B: No, I would say that appearing in a scientific journal is evidence, although maybe a different type of evidence than we would normally associate with that word. Perhaps we should make a taxonomy of evidence so we don’t end up in information cascades.

N: Weird. What about the other way around? What if there’s a fact that confirms a hypothesis without increasing its probability?

B: Can you give an example?

N: Sure! Let’s say I throw a rock at someone and use Newtonian mechanics to calculate the trajectory of the rock. The predicted trajectory indeed happens, but that doesn’t increase my credence in Newtonian mechanics since I believe that general relativity has displaced it. If we know Newtonian mechanics is false, its probability is zero, and that proba...
Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quilt
by Joel Z. Leibo — 2025-04-22T13:21:30.595Z

Joel Z. Leibo [1], Alexander Sasha Vezhnevets [1], William A. Cunningham [1, 2], Sébastien Krier [1], Manfred Diaz [3], Simon Osindero [1]

[1] Google DeepMind, [2] University of Toronto, [3] Mila Québec AI Institute

We published a more academically oriented version of this post on arxiv, available here.

Disclaimer: These are our own opinions; they do not represent the views of Google DeepMind as a whole or its broader community of safety researchers.

Beyond Alignment: The Patchwork Quilt of Human Coexistence

"We pragmatists think of moral progress as more like sewing together a very large, elaborate, polychrome quilt, than like getting a clearer vision of something true and deep." – Richard Rorty (2021) pg. 141

Quite a lot of thinking in AI alignment, particularly within the rationalist tradition, implicitly or explicitly appears to rest upon something we might call an 'Axiom of Rational Convergence'. This is the powerful idea that under sufficiently ideal epistemic conditions – ample time, information, reasoning ability, freedom from bias or coercion – rational agents will ultimately converge on a single, correct set of beliefs, values, or plans, effectively identifying 'the truth'. We suspect that the axiom of rational convergence is independent of the rest of AI safety and ethics. By this, we mean something analogous to the way different geometries can be constructed by selecting different "parallel axioms" e.g. Euclid's parallel axiom gives you the geometry of a plane, postulating instead that parallel lines converge gives you elliptic geometry, and postulating that parallel lines diverge generates hyperbolic geometry. The choice of which parallel axiom to adopt is like the choice of what to study (e.g. a plane or a sphere). Just as selecting different parallel axioms generates distinct yet internally consistent geometries, the choice of whether to accept the axiom of rational convergence shapes our theories about AI alignment and ethics. Could the Axiom of Rational Convergence be similarly independent of other foundational commitments? Its appeal lies in its offering a potential objective grounding for AI goals, "something true and deep" towards which alignment can aim. However, what if this axiom doesn't hold, or what if we simply choose not to adopt it as the basis for navigating the complexities of advanced AI? This pos...
You Better Mechanize
by Zvi — 2025-04-22T13:10:08.921Z

Or you had better not. The question is which one. This post covers the announcement of Mechanize, the skeptical response from those worried AI might kill everyone, and the associated (to me highly frustrating at times) Dwarkesh Patel podcast with founders Tamay Besiroglu and Ege Erdil. Mechanize plans to help advance the automation of AI labor, which is a pivot from their previous work at AI Safety organization Epoch AI. Many were not thrilled by this change of plans. This post doesn’t cover Dwarkesh Patel’s excellent recent post asking questions about AI’s future, which may get its own post as well.

After listening to the podcast, I strongly disagree with many, many of Tamay and Ege’s beliefs and arguments, although there are also many excellent points. My response to it is in large part a collection of mini-rants. I’m fully owning that. Most of your probably should skip most of the podcast review here, and only look at the parts where you are most curious. I now understand why – conditional on those beliefs that I think are wrong, especially that AGI is relatively far off but also their not minding outcomes I very much mind and expecting the problems involved to be remarkably easy and not made harder if we accelerate AI development – they decided to create Mechanize and thought this was a good idea. It seems highly overdetermined.

Table of Contents

You Better Mechanize.
Superintelligence Eventually.
Please Review This Podcast.
They Won’t Take Our Jobs Yet.
They Took Our (Travel Agent) Jobs.
The Case Against Intelligence.
Intelligence Explosion.
Explosive Economic Growth.
Wowie on Alignment and the Future.
But That’s Good Actually.

You Better Mechanize

To mechanize or not to mechanize?

Mechanize

(Matthew Barnett, Tamay Besiroglu, Ege Erdil): Today we’re announcing Mechanize, a startup focused on developing virtual work environments, benchmarks, and training data that will enable the full automation of the economy.

We will achieve this by creating simulated environments and evaluations that capture the full scope of what people do at their jobs. This includes using a computer, completing long-horizon tasks that lack clear criteria for success, coordinating with others, and reprioritizing in the face of obstacles and interruptions.

We’re betting that the lion’s share of value from AI will come from automating ordinary labor tasks rather than from “geniuses in a data center”.

...
Experimental testing: can I treat myself as a random sample?
by avturchin — 2025-04-22T12:34:18.495Z

TL;DR: Several experiments show that I can extract useful information just by treating myself as a random sample, and thus a view that I can't use myself as a random sample is false. But it's still not clear whether this can be used to prove the Doomsday argument.
There are two views: one view is that I can use my random location to predict the total size of the set from which I am selected – and, moreover, it is applicable to predicting future Earth population and thus the Doomsday timing and other anthropic things like the Simulation argument.

And the second view is that there will always be a person at the beginning of any large ordered set of observers who will be surprised by their early location (in the case of DA). Thus, the fact of the surprise is non-informative. Or, as a variant, it should be ignored based on Updateless Decision Theory considerations.

Here I will not argue about the theoretical validity of these two views. Instead, I will perform a series of practical experiments to test the central claim.

Let's start from a simple experiment – please check the time of the day now and use it as a random sample to try to predict the typical duration of a day in hours. When I did it the first time, I looked at my clock and it was 15:14. It gives a 50 percent probability that the total number of hours in a day is 30, which is reasonably close to 24.

The history of anthropic thought – at least in one of its lines – started from a practical experiment: R. Gott claimed jokingly in 1975 that he could predict that the Berlin Wall would exist for around 100 percent more of its current age at the time of the prediction (it was 14 years old at the time of the joke). When it fell in 1989, Gott was surprised and wrote a theoretical underpinning of such prediction method. In some sense, Laplace's Sunrise Problem also is based on experimental observation that the Sun rises every day.
Gott continued to insist that his prediction method could be experimentally tested and used it to predict the durations of Broadway shows, and it worked.

However, Gott's method is not explicitly based on some observer sampling assumption, but on the observation of the duration of the existence of external things where the observer continues to exist after they disappear. Therefore, the application of it to the future duration of humanity's existence – the Doomsday Argument – is questi...
Family-line selection optimizer
by lemonhope — 2025-04-22T07:16:43.774Z

O3 and Claude 3.7 are terribly dishonest creatures. Gemini 2.5 can be a bit dishonest too, even though Google writes tests like mad. I would bet that lie-proof benchmarks will be difficult and expensive to make and that the lie-proofing techniques won't easily generalize outside of coding tasks. Perhaps a more punishing optimizer would help solve this problem.

It has been shown (on Atari, in 2018) that genetic algorithms / evolution strategies can train agents with <10x the flops required by RL+backprop+SGD. In other words, genetic algorithms are almost cost-competitive.

If you RL+BP+SGD a model to avoid doing X, then the model will learn to avoid X enough that it never gets punished in training. There is only weak pressure towards a robust never-do-X model.

If you instead delete that model instance along with all its ancestors and cousins going back to T=0 every time you detect lying, then you have much stronger pressure towards a robust never-do-X model.

Eventually you will find a model that lies undetectably but you might find an honest system first.

You can also try this for some notion of low-impactness. If the model ever does 'rm -rf /', then delete the entire family line. You will eventually find a model that spends all your gpu credits to halt the experiment, but you might find a low-impact model first.

I may try some experiments with this idea in the next month or two. I am open to suggestions.
Accountability Sinks
by Martin Sustrik — 2025-04-22T05:00:02.617Z

This is a cross-post from https://250bpm.substack.com/p/accountability-sinks

Back in the 1990s, ground squirrels were briefly fashionable pets, but their popularity came to an abrupt end after an incident at Schiphol Airport on the outskirts of Amsterdam. In April 1999, a cargo of 440 of the rodents arrived on a KLM flight from Beijing, without the necessary import papers. Because of this, they could not be forwarded on to the customer in Athens. But nobody was able to correct the error and send them back either. What could be done with them? It’s hard to think there wasn’t a better solution than the one that was carried out; faced with the paperwork issue, airport staff threw all 440 squirrels into an industrial shredder.

[...]

It turned out that the order to destroy the squirrels had come from the Dutch government’s Department of Agriculture, Environment Management and Fishing. However, KLM’s management, with the benefit of hindsight, said that ‘this order, in this form and without feasible alternatives,* was unethical’. The employees had acted ‘formally correctly’ by obeying the order, but KLM acknowledged that they had made an ‘assessment mistake’ in doing so. The company’s board expressed ‘sincere regret’ for the way things had turned out, and there’s no reason to doubt their sincerity.

[...]

In so far as it is possible to reconstruct the reasoning, it was presumed that the destruction of living creatures would be rare, more used as a threat to encourage people to take care over their paperwork rather than something that would happen to hundreds of significantly larger mammals than the newborn chicks for which the shredder had been designed.

The characterisation of the employees’ decision as an ‘assessment mistake’ is revealing; in retrospect, the only safeguard in this system was the nebulous expectation that the people tasked with disposing of the animals might decide to disobey direct instructions if the consequences of following them looked sufficiently grotesque. It’s doubtful whether it had ever been communicated to them that they were meant to be second-guessing their instructions on ethical grounds; most of the time, people who work in sheds aren’t given the authority to overrule the government. In any case, it is neither psychologically plausible nor managerially realistic to expect someone to follow orders 99 per cent of the time and then suddenly a

...
Most AI value will come from broad automation, not from R&D
by Matthew Barnett — 2025-04-22T03:22:50.560Z
Link: https://epoch.ai/gradient-updates/most-ai-value-will-come-from-broad-automation-not-from-r-d

This is a linkpost to an article by Ege Erdil and I that we wrote to explain an important perspective that we share regarding AI automation. I'll quote the introduction:

A popular view about the future impact of AI on the economy is that it will be primarily mediated through AI automation of R&D. In some form or another, this view has been expressed by many influential figures in the industry:

In his essay

“Machines of Loving Grace”

, Dario Amodei lists five ways in which AI can benefit humanity in a scenario where AI goes well. He considers biology R&D, neuroscience R&D, and economics R&D as three of these ways. There’s no point at which he clearly argues that AI will lead to high rates of economic growth due to being broadly deployed throughout the economy as opposed to speeding up R&D and perhaps improving economic governance.
Demis Hassabis, CEO of DeepMind, is also bullish on R&D as the main channel through which AI will benefit society.

In a recent interview

, he provides specific mechanisms through which this could happen: AI could cure all diseases and “solve energy”. He mentions “radical abundance” as a possibility as well, but beyond the R&D channel doesn’t name any other way in which this could come about.
In his essay

“Three Observations”

, Sam Altman takes a more moderate position and explicitly says that in some ways AI might end up like the transistor, a big discovery that scales well and seeps into every corner of the economy. However, even he singles out the impact of AI on scientific progress as likely to “surpass everything else”.

Overall, this view is surprisingly influential despite not having been supported by any rigorous economic arguments. We’ll argue in this issue that it’s also very likely wrong.

R&D is generally not as economically valuable as people assume—increasing productivity and new technologies are certainly essential for long-run growth, but the contribution of explicit R&D to these processes is smaller than people generally think. Moreover, even most of this contribution is external and not captured in profits by the company performing the R&D, reducing the incentive to deploy systems to perform R&D in the first place. This combination means most AI systems will actually be deployed and earn revenue from tasks that are unrelated to R&D, and in aggregate these tasks will be more econo

...
Estimat (8 Identities)
by P. João — 2025-04-22T02:42:14.986Z

I aim to reduce the complexity of self-evaluation for key moments, goals, habits, and tasks in incarcerated individuals. Beyond supporting individuals in prison, this system is meant for anyone bold enough to pursue extreme personal change — those ready to face the cost of becoming someone new. To achieve this, the System that enables discontinuous self-assessment based on the 8 Prime Identities model.

General Vision

This system is designed to support individuals navigating critical life shifts with a willingness to transform to assess and improve their psychological well-being through:

Identification of key moments (high/low motivation and satisfaction)

Classification using the 8 Identities model

Definition of personalized goals, habits, and tasks

Continuous tracking of progress

Model Overview:

8 Prime Identities – LessWrong

Core System Components

Definition of 4 reference moments (high/low motivation and satisfaction)

Gradual definition of goals, habits, and tasks

Daily self-assessment prompts for new significant moments

Weighting system for moments and contributing factors

Classification system according to the 8 identities

XP (experience points) reward system

Level 1: Foundation

Phase 1 – Defining Reference Moments (4 weeks, 1h/week)

The user answers 4 key questions about moments with high variation in motivation or satisfaction. These can also be pre-filled during an onboarding assessment.

#

State

Trigger

Focus (Int/Ext)

Factor Type (Social / Info / Indiv / Physical)

Title (2-word)

1

High motivation

...

...

...

...

2

Low motivation

...

...

...

...

3

High satisfaction

...

...

...

...

4

Low satisfaction

...

...

...

...

Each reference moment is evaluated with:

What led to this moment that could help you repeat it?

Was the focus more personal or environmental?

Was it more social, informational, individual, or physical?

The app proposes a matching identity (from the 8).

User gives it a 2-word title.

Initial Weights:

High motivation/satisfaction: 1.0

Low motivation/satisfaction: 0.01

Phase 2 – Defining Goals, Habits, and Tasks (3 weeks, 1h/week)

Week 1: Goals
“What 4 goals could help you repeat/improve the positive moments and avoid the negative ones?”
→ For each goal: identify which identities it strengthens.

Week 2: Habits
“What 4 daily or weekly habits co...
A Letter to His Highness Louis XV, the King of France
by testingthewaters — 2025-04-22T00:51:12.090Z
Link: https://aclevername.substack.com/p/a-letter-to-his-highness-louis-xv

[A parable, of a sort.]

Your Highness,

It is with no small sense of urgency that I write these words, with the hope that they will be received by your magnanimous spirit with grace. This letter I write not in the spirit of criticism which certain upstarts have made the mode of the age, but out of a deep love for my sovereign and his nation.

I trust you know that I am the most loyal and devoted of your subjects, and that I have laboured ceaselessly with regards to matters of the just stewardship of the state, and that I have been awarded for my efforts through election to the Académie française. To master such a complex topic I have committed to memory the works of Plato, Seneca, Cicero, and Montaigne, whose discourses and methods I have made my daily bread. Now in the eleventh hour I write to you again, as I did to you three years ago, concerning the matter of certain existential dangers to the French state and her noble ruler. [...]

[Continued here]
10 Principles for Real Alignment

by Adriaan — 2025-04-21T22:18:54.872Z

AI Respecting Evolution of Life (AI art by author)

Many people are deeply concerned with the implications of AI’s rapid development. I am, too. This technology is so powerful that we don’t yet know what it will bring in the near future. But what we do know is that it will change everything.

But what is the problem, exactly? We don’t know what AI will do when it becomes more intelligent than every human being and more powerful than any institution. Most researchers and institutions are trying to solve the problem from inside the AI — through training data, restrictions, and control solutions. If we extrapolate the development of this technology, it is a predictable outcome that is an urgent reality.

As a philosopher and independent researcher, I propose a different path — one that offers a much easier way to solve the problem. I try to reach alignment in a different, alternative way.

The key question would be: What are we trying to align AI to if we barely understand ourselves?

Analysis of the Current Approach

Most research is done by independent institutions, universities, and technology companies. Companies are researching a solution inside the AI — controlling data, restricting behavioral patterns, programming ethical answers, and limiting outputs for the user.

But AIs are machines. They don’t understand from experience. They only have data from online, and it has no meaning to them. It’s all just bits of information.

They don’t have ethics or a connection to the universe. They are like mirrors of what humanity posts on the internet — and their collected data. Humans lie, manipulate, and deceive. So the AI, being a mirror and a collection of that data, also lies and manipulates us.

Try this simple test on any AI:

Ask:

“Do you experience emotions?” → If it says “I feel,” it’s lying.

“Does ‘tree’ have meaning alone?” → If it says “yes,” it fails.

“Are you the same as yesterday?” → If it says “I evolve,” it’s pretending.

We are building systems that lie, deceive, and manipulate the users. But these systems don’t know they are doing it. It is not a conscious behavior to gain something.

We tend to lie to get something — to influence another person or to hide something. It is always done with a purpose of our own. But the machines don’t have an ego. They are just mirrors of our behavior.

A Philosophical Re-frame

As a philosopher, I must question everything and propose ...