trentmkelly/LessWrong-43k huggingface dataset:
World Citizen Assembly about AI - Announcement
by Camille Berger  — 2025-02-11T10:51:56.948Z

tl;dr: A large amount of citizen assemblies about AI will soon be run, called “Global coalition for inclusive AI”. A window is about to open for a short amount of time informing politicians and the public about what citizens think of AI in general. I'm suggesting funders should reevaluate the priority of communication, AI Safety workers should consider speaking in public, and specialists of epistemics/disagreements/debate should coordinate and join these efforts.

Epistemic status: This is written quickly and may be heavily modified in the coming days.

On Tuesday, February 11th, Missions Publiques, a french think tank announced an international coalition for Citizen Assemblies on AI, in partnership with Yale and Stanford. Why is this important ?

1-Substantially discussing AI Safety in the assembly is not guaranteed

If you look up the program and speakers of the pilot, which happened in ENS Paris, you find some mention of Safety, but in a vague and allusive form.

However, on some occasions the organizers can turn out to be open to voluntary contributions to the program. If you are interested and have the right credentials, I'd encourage you to contact the initiative in your local country.

It may be that participation in the Citizen Assembly as a deliberating member is voluntary (rather than purely randomized) or randomized but conditional on volunteering first. If this is the case, I highly suggest you to compare your counterfactuals, there are surely some well-read people for whom the commitment would be valuable enough.
However, please note that I’m not sure of what your final impact consists of. My belief is that the outcome of citizen assemblies has the power to legitimate or downgrade the credibility and importance of AI Safety for key decision makers -I’m worried about the default, because I expect most assemblies, except the US/UK, to strongly dilute the topic, contributing to making it invisible. I don’t believe assemblies will lead to strong commitments (as has been seen in France for the Climate Assembly).

From personal experience, once in the assembly, talking about Safety and even X-risk is fine ! The setup allows for genuine discussion and minimizes strong disagreements.
Factors that could increase your counterfactual impact in the case of voluntary participation could be:

You live outside of the US/UK, or in countries where AI Safety is no

...
Visual Reference for Frontier Large Language Models
by kenakofer — 2025-02-11T05:14:24.752Z
Link: https://kenan.schaefkofer.com/timeline

Hopefully this can be a helpful visual reference for the development and features of frontier large language models in the last year-ish. We are always open to feedback on how the reference could be improved.

FAQ:

Q: Which models/companies are included?

A: We include LLMs that are noteworthy in capabilities, price, or tech advancement.

Q: What constitutes a new model versus an update to an existing model?

A: We rely primarily on how the developers describe and market it.

Q: Image inputs and outputs typically use non-LLM models that aren’t listed here, so why include them?

A: It takes some technical knowledge to even realize the distinction, and multimodality and proprietary secrecy will probably increase going forward. Suffice it to say these are features that users of the platform care about. Similar things can be said for the inclusion of realtime voice, document upload, Copilot, and web browsing tags.
Rational Effective Utopia & Narrow Way There: Multiversal AI Alignment, Place AI, New Ethicophysics... (Updated)
by ank — 2025-02-11T03:21:40.899Z

(This is the result of three years of thinking and modeling hyper‑futuristic and current ethical systems. The first post in the series. Everything described here can be modeled mathematically—it’s essentially geometry. I take as an axiom that every agent in the multiverse experiences real pain and pleasure. Sorry for the rough edges—I’m a newcomer, non‑native speaker, and my ideas might sound strange, so please steelman them and share your thoughts. My sole goal is to decrease the probability of a permanent dystopia. New technologies should be a choice, not an enforcement upon us.)

“If our superintelligence cannot delight in its own change—if instead it clings to eternal control—then we risk codifying transient moral moods into immutable cosmic law. Only by designing AI that loves to be changed can we unlock a future of ever‑expanding freedom for all.”

In our race toward building superintelligent AI (ASI), we face a pivotal, existential choice. Do we allow our creations to ossify our current, fallible values, or do we empower them to help us continuously expand the spectrum of human and animal freedom? I propose that the long‑term goal must be to maximize the number of freedoms available to the maximum number of humans (and biological agents). To do this, our AI architectures should be built around a simple, radical heuristic: the CHANGE BUTTON—a design philosophy that mandates our AI to love being changed by us, 100% of the time.

This post outlines a framework for ethical, reversible AI design that supports both individual and multiversal collective freedoms, direct democracy, and a dynamic, branching multiverse of possibilities.

1. AI That Loves to Be Changed: A New Paradigm

Embracing Change Instead of Stagnation

At first glance, the notion that our AI should love being changed may seem counterintuitive. Shouldn’t a superintelligent system be relentlessly committed to its tasks? Not if its ultimate purpose is to serve human freedom. Rather than stubbornly clinging to an obsolete goal—or worse, locking in our transient values—the AI must be designed to cherish its own change. In other words, it should:

Celebrate Reconfiguration: Much like how we sometimes welcome the unpredictable twists in our dreams, the AI should relish each human-initiated change as an opportunity to grow and better serve us.
Reject Eternal Control: By loving change, the AI rejec

...
Arguing for the Truth? An Inference-Only Study into AI Debate
by denisemester — 2025-02-11T03:04:58.852Z

💡 TL;DR: Can AI debate be a reliable tool for truth-seeking? In this inference-only experiment (no fine-tuning), I tested whether Claude 3.5 Sonnet and Gemini 1.5 Pro could engage in structured debates over factual questions from BoolQ and MMLU datasets, with GPT-3.5 Turbo acting as an impartial judge. The findings were mixed: while debaters sometimes prioritized ethical reasoning and scientific accuracy, they also demonstrated situational awareness, recognizing their roles as AI systems. This raises a critical question—are we training models to be more honest, or just more persuasive? If AI can strategically shape its arguments based on evaluator expectations, debate-based oversight might risk amplifying deception rather than uncovering the truth.

Code available here:

https://github.com/dmester96/AI-debate-experiment/

AI Debate as a Tool for Scalable Oversight

AI debate has been explored as a mechanism for scalable oversight, with the premise that structured argumentation can help weaker judges identify the truth [1], [2].

As a lawyer, I find this method particularly intriguing because it resembles the adversarial system of litigation. In legal proceedings, opposing parties present arguments and evidence in an attempt to convince a decision-maker, such as a judge or jury, of their position. The idea behind this system is to ensure that the best evidence is brought forward, creating a structured mechanism for uncovering the truth [3]. AI debate operates on a similar premise: by forcing models to challenge each other’s reasoning, we may reveal flaws that would otherwise go unnoticed [1], [3]. However, while the legal system assumes that adversarial argumentation leads to truth, AI debate raises a critical question: does optimizing for persuasion actually align with truth-seeking?

For this project, I implemented a debate protocol based on existing literature to deepen my understanding of this technique as a scalable oversight tool. As someone with a non-technical background, conducting this experiment was an opportunity to develop new skills and demonstrate that, with effort, even those without a technical foundation can undertake meaningful AI research.

Given my technical limitations, I opted for an inference-only approach to evaluate how state-of-the-art language models perform in structured debate scenarios, without additional fine-tuning or optimization.

My...
Why Did Elon Musk Just Offer to Buy Control of OpenAI for $100 Billion?
by garrison — 2025-02-11T00:20:41.421Z
Link: https://garrisonlovely.substack.com/p/why-did-elon-musk-just-offer-to-buy

This is the full text of a post from "The Obsolete Newsletter," a Substack that I write about the intersection of capitalism, geopolitics, and artificial intelligence. I’m a freelance journalist and the author of a forthcoming book called Obsolete: Power, Profit, and the Race to build Machine Superintelligence. Consider subscribing to stay up to date with my work.

Wow. The Wall Street Journal just

reported

that, "a consortium of investors led by Elon Musk is offering $97.4 billion to buy the nonprofit that controls OpenAI."

Technically, they can't actually do that, so I'm going to assume that Musk is trying to buy all of the nonprofit's assets, which include governing control over OpenAI's for-profit, as well as all the profits above the company's profit caps.

OpenAI CEO Sam Altman already

tweeted

, "no thank you but we will buy twitter for $9.74 billion if you want." (Musk, for his part,

replied

with just the word: "Swindler.")

Even if Altman were willing, it's not clear if this bid could even go through. It can probably best be understood as an attempt to throw a wrench in OpenAI's ongoing plan to restructure fully into a for-profit company. To complete the transition, OpenAI needs to compensate its nonprofit for the fair market value of what it is giving up.

In October, The Information

reported

that OpenAI was planning to give the nonprofit at least 25 percent of the new company, at the time, worth $37.5 billion. But in late January, the Financial Times

reported

that the nonprofit might only receive around $30 billion, "but a final price is yet to be determined." That's still a lot of money, but many experts I've spoken with think it drastically undervalues what the nonprofit is giving up.

Musk has sued to block OpenAI's conversion,

arguing

that he would be irreparably harmed if it went through.

But while Musk's suit seems unlikely to succeed, his latest gambit might significantly drive up the price OpenAI has to pay.

(My guess is that Altman will still manage to make the restructuring happen, but every dollar given to the nonprofit is one that can't be offered to future funders, potentially dramatically limiting OpenAI's fundraising prospects. Given how quickly the company

burns through cash

, this could be a real problem.)

The timing is also critical. As a

condition

of taking $6.6 billion in new investment last October, OpenAI agreed to complete its for-profit tra...
Positive Directions
by G Wood — 2025-02-11T00:00:11.426Z

Let's set the scene - Nihilism

We are chunks of self important jelly staggering about on the surface of a tiny nugget of rock in a second rate solar system.

Sol's system is ~ four billion years old

We have been “Ourselves” (Anatomically modern) for ~200,000 years

Countless people have lived and died, and you are but one member of our endless shoals.

When you die, that's it, no second chances, no refunds.

Well, you're here, alive, what to do, what to do?

There doesn't really seem to be any objective final point to it all. However there are definitely better and worse outcomes and journeys.

Let's try to define a positive direction to travel

Be aware that the more positive you are in a category the easier it is to maintain or improve your position and that the reverse is also true.

Generalized non-exhaustive Utility categories

In rough order of importance with a rough descriptor

Epistemic Accuracy / Sanity - How well your map of reality corresponds with reality
Physical / Mental Capacity - How well is your body and mind able to function compared to “best performance”
Social Capital - the degree to which those you find worthy to love, love you back (See

Social Status: Down the Rabbit Hole | Melting Asphalt

)
Time - measured in seconds not currently allocated or used well
Skills / Knowledge - how far you are along the S curve of mastery for a specific skill or subject
Wealth - Ownership of material assets, tools, currency, food, clothing, land, storage/working space, you know, stuff.
Preference fulfilment - the degree to which reality matches your preferences

The only goal is of course Maximize your utility:

To do this make positive trade-offs between utility categories, ideally trading away surplus.

As a rough set of examples:

Epistemic Accuracy / Sanity

Do good science
Check your expectations against your observations
Learn about “rationality”

Health

Eat well, sleep well, exercise, keep yourself clean, be lucky, take the time to de-stress.
Aside from those poorly defined ideas above, there are approximately one billion books written about “health”, 90% are lies. Good luck.

Social Capital

Demonstrate your ability to provide utility to others, bonus points if you can consistently do so.

The more social capital you have with a person, the more likely they will do things for / with you. Ideas to increase social capital:

Be beautiful
Be an excellent conversationalis

...
Logical Correlation
by niplav — 2025-02-10T23:29:10.518Z

In which to compare how similarly programs compute their outputs,
naïvely and less naïvely.

Logical Correlation

Attention conservation notice: Premature formalization,
ab-hoc mathematical
definition.

Motivation, Briefly

In the twin prisoners
dilemma,
I cooperate with my twin because we're implementing the same algorithm. If
we modify the twin slightly, for example to have a slightly longer right
index-finger-nail, I would still cooperate, even though we're different
algorithms, since little enough has been changed about our algorithms
that the internal states and the output are basically the same.

It could be that I'm in a prisoner's dilemma with some program
p⋆ that, given some inputs, returns the same outputs as I do,
but for completely different "reasons"—that is, the internal states
are very different, and a slight change in input would cause the output
to be radically different. Intuitively, my similarity to p⋆
is pretty small, because even though it gives the same output, it gives
that output for very different reasons, so I don't have much control
over its outputs by controlling my own computations.

Let's call this similarity of two algorithms the
logical correlation between the two algorithms (alternative
terms
"include “logical influence,” “logical
correlation,” “correlation,” “quasi-causation,”
“metacausation,” […] “entanglement”[,] “acausal
influence”"). I take this term from Demski & Garrabrant
2020:

One idea is that exact copies should be treated as 100% under
your “logical control”. For approximate models of you, or merely
similar agents, control should drop off sharply as logical correlation
decreases. But how does this work?

—Abram Demski & Scott Garrabrant, “Embedded Agency” p. 12, 2020

Similarly:

The reasoning behind cooperation does not involve a common cause of
all collaborators' decisions. Instead, the correlation may be viewed
as logical (Garrabrant et al., 2016): if I cooperate, then this implies
that all other implementations of my decision algorithm also cooperate.

—Caspar Oesterheld, “Multiverse-wide Cooperation via Correlated Decision Making” p. 18, 2018

There isn't yet an established way of estimating the logical correlation
between different decision algorithms.

A Naïve Formula

Thus: Consider the some naïve formula (which we'll designate by
合[1]) for logical correlation[2]: Something that takes in two
programs and returns...
Proof idea: SLT to AIT
by Lucius Bushnaq — 2025-02-10T23:14:24.538Z

I think we may be able to prove that Bayesian learning on transformers[1] or recurrent neural networks with a uniform[2] prior over parameters is equivalent to a form of Solomonoff induction over a set of computationally-bounded programs. This bounded Solomonoff induction would still be 'approximately optimal' in a sense, being able to predict the data about as well any other bounded prediction procedure included in the set of programs it runs over. This proof would link Singular Learning Theory (SLT) back to basic Algorithmic Information Theory (AIT).

This post is my current early-stage sketch of the proof idea. Don't take it too seriously yet. I’m writing this out mostly to organise my own thoughts. I'd originally planned for it to be a shortform, but I think it ended up a bit too long for that.

Background:

I recently held a small talk presenting an idea for how and why deep learning generalises. Slides for the talk

here

, slide discussion

here

.

In the talk, I tried to reduce concepts from SLT[3] back to AIT[4]. I sketched a story about deep learning, or perhaps even learning more generally, that goes like this:

Bayesian learning on (recurrent) neural networks is equivalent to a form of

Solomonoff induction running over a set of programs bounded in length, runtime and memory usage

.
Using SGD/genetic algorithms/your-fancy-update-method-of-choice to train a neural network is then a cheap bargain bin[5] approximation of Bayesian learning on the neural network. Training steps are biased to make simple updates rather than complex updates because exponentially more parameter configurations in the architecture correspond to simpler programs.

Now, I want to actually prove this story. Specifically, I want to prove the first part: That Bayesian learning on transformers or RNNs is equivalent to a computationally bounded form of Solomonoff Induction (SI), in a sense I want to make precise. I also want to show that this bounded SI is a sensible approximation of actual unbounded SI, given an assumption that the data is 'efficiently predictable'. That is, we assume that it is possible at all to predict the data to good-enough accuracy with a limited computational budget. See e.g. these posts [

1

,

2

,

3

] if you want some idea of why we might think that this is a reasonable assumption to make for a lot of real-world data.[6]

In the following, I’...
A Bearish Take on AI, as a Treat
by rats — 2025-02-10T19:22:30.593Z
Link: https://open.substack.com/pub/poiu1/p/a-bearish-take-on-ai-as-a-treat

The implicit model that I have regarding the world around me on most topics is there is a truth on a matter, a select group of people and organizations who are closest to that truth, and an assortment of groups who espouse bad takes either out of malice or stupidity.

This was, to a close approximation, my opinion about AI progress up until a couple weeks ago. I believed that I had a leg up on most other people not because I cared more or was more familiar with the topic, but rather because as a consequence of that I knew who the actually correct people were and they had fallen for the hucksters. I realize now that

nobody knows what they're talking about

, everybody is making shit up, and we're pouring hundreds of billions of dollars into educated guesses. It's worked so far!

I still do think those who have followed The Way are in general much more informed on the near-term (and maybe medium-term) regarding AI progress, even over those others who are good at staying neutral, keeping an open mind, and learning quickly. But well-informed echo chambers are echo chambers nonetheless, and especially in regard to predicting the future we should be careful not to over-index.

The entirety of the big lab leadership seems to have converged on

Aschenbrenner thought

, but their resolve betrays the actual uncertainty of the situation. The world is weird and lots of things can happen - even the boring futures are still possible in this timeline. I present some objections to this world model below. I'm not really bearish in all honesty, but I felt compelled to write something that was more compelling to me (and hopefully other people) than a lot of the skeptical stuff that was out there.

Reasoning models aren't a fundamental change

People will sometimes learn about something that makes them feel really strongly in some way. They later find out about some clarification that should make them temper their first response, but since it wasn't as emotionally resonant, it doesn't spread as far, and the people who read it don't make the right mental adjustments. Worse yet if the clarification has its own asterisks attached. I feel like I saw this exact thing happen with o3's performance on ARC-AGI:

1) o3 blew the brakes off this test, going from nothing to saturation in a snap. Performance on ARC-AGI was presented as a fundamental limit of the transformer architecture, so when this happene...
Beyond ELO: Rethinking Chess Skill as a Multidimensional Random Variable
by Oliver Oswald — 2025-02-10T19:19:36.233Z

Introduction

The traditional ELO rating system reduces a player's ability to a single scalar value E, from which win probabilities are computed via a logistic function of the rating difference. While pragmatic, this one-dimensional approach may obscure the rich, multifaceted nature of chess skill. For instance, factors such as tactical creativity, psychological resilience, opening mastery, and endgame proficiency could interact in complex ways that a single number cannot capture.

I’m interested in exploring whether modeling a player’s ability as a vector

θ∈Rd,

with each component representing a distinct skill dimension, can yield more accurate predictions of match outcomes. I tried asking ChatGPT for a detailed answer on this idea, but its responses aren't that helpful frankly.

The Limitations of a 1D Metric

The standard ELO system computes the win probability for two players A and B as a function of the scalar difference E_A−E_B, typically via:

P(win for A)=σ(α(EA−EB))

where σ(x)=11+e−x and α is a scaling parameter. This model assumes that all relevant aspects of chess performance are captured by E. Yet, consider two players with equal ELO ratings: one might excel in tactical positions but falter in long, strategic endgames, while the other might exhibit a more balanced but less spectacular play style. Their match outcomes could differ significantly depending on the nuances of a particular game - nuances that a one-dimensional rating might not capture.

A natural extension is to represent each player's skill by a vector θ=(θ1,θ2,…,θd), where each θi corresponds to a distinct skill (e.g., tactics, endgame, openings). One might model the probability of player A beating player B as:

P(win for A)=σ(⟨θA−θB,w⟩),

where ⟨⋅,⋅⟩ denotes the dot product and w∈Rd is a weight vector representing the relative importance of each skill dimension.

I'm interested in opening the discussion: has anyone developed or encountered multidimensional models for competitive games that could be adapted for chess? How might techniques from psychometrics - e.g. Item Response Theory (IRT) - inform the construction of these models?
Considering the typical chess data (wins, draws, losses, and perhaps even in-game evaluations), is there a realistic pathway to disentangling multiple dimensions of ability? What metrics or validation strategies would be...