trentmkelly/LessWrong-43k huggingface dataset:
The Risk of Gradual Disempowerment from AI
by Zvi — 2025-02-05T22:10:06.979Z

The baseline scenario as AI becomes AGI becomes ASI (artificial superintelligence), if nothing more dramatic goes wrong first and even we successfully ‘solve alignment’ of AI to a given user and developer, is the ‘gradual’ disempowerment of humanity by AIs, as we voluntarily grant them more and more power in a vicious cycle, after which AIs control the future and an ever-increasing share of its real resources. It is unlikely that humans survive it for long.

This gradual disempowerment is far from the only way things could go horribly wrong. There are various other ways things could go horribly wrong earlier, faster and more dramatically, especially if we indeed fail at alignment of ASI on the first try.

Gradual disempowerment it still is a major part of the problem, including in worlds that would otherwise have survived those other threats. And I don’t know of any good proposed solutions to this. All known options seem horrible, perhaps unthinkably so. This is especially true is the kind of anarchist who one rejects on principle any collective method by which humans might steer the future.

I’ve been trying to say a version of this for a while now, with little success.

Table of Contents

We Finally Have a Good Paper.

The Phase 2 Problem.

Coordination is Hard.

Even Successful Technical Solutions Do Not Solve This.

The Six Core Claims.

Proposed Mitigations Are Insufficient.

The Social Contract Will Change.

Point of No Return.

A Shorter Summary.

Tyler Cowen Seems To Misunderstand Two Key Points.

Do You Feel in Charge?.

We Will Not By Default Meaningfully ‘Own’ the AIs For Long.

Collusion Has Nothing to Do With This.

If Humans Do Not Successfully Collude They Lose All Control.

The Odds Are Against Us and the Situation is Grim.

We Finally Have a Good Paper

So I’m very happy that Jan Kulveit*, Raymond Douglas*, Nora Ammann, Deger Turan, David Krueger and David Duvenaud have taken a formal crack at it, and their attempt seems excellent all around:

AI risk scenarios usually portray a relatively sudden loss of human control to AIs, outmaneuvering individual humans and human institutions, due to a sudden increase in AI capabilities, or a coordinated betrayal.

However, we argue that even an incremental increase in AI capabilities, without any coordinated power-seeking, poses a substantial risk of eventual human disempowerment.

This loss of human influence wi

...
Wired on: "DOGE personnel with admin access to Federal Payment System"
by Raemon — 2025-02-05T21:32:11.205Z
Link: https://web.archive.org/web/20250205123049/https://www.wired.com/story/elon-musk-associate-bfs-federal-payment-system/

I haven't looked into this in detail, and I'm not actually sure how unique a situation this is. But, it updated me on "institutional changes to the US that might be quite bad[1]", and it seemed good if LessWrong folk did some sort of Orient Step on it.

(Please generally be cautious on LessWrong talking about politics. I am interested in people commenting here who have read the LessWrong Political Prerequisites sequence. I'll be deleting or at least unhesitatingly strong downvoting comments that seem to be doing unreflective partisan dunking)

((But, that's not meant to mean "don't talk about political actions." If this is as big a deal as it sounds, I want to be able to talk about "what to do do?". But I want that talking-about-it to feel more like practically thinking through an action space, than blindly getting sucked into a political egregore))

A 25-year-old engineer named Marko Elez, who previously worked for two

Elon Musk companies

, has direct access to Treasury Department systems responsible for nearly all payments made by the

US government

, three sources tell WIRED.

Two of those sources say that Elez’s privileges include the ability not just to read but to write code on two of the most sensitive systems in the US government: the Payment Automation Manager and Secure Payment System at the Bureau of the Fiscal Service (BFS). Housed on a secure mainframe, these systems control, on a granular level, government payments that in their totality amount to more than a fifth of the US economy.

Despite

reporting that suggests

that Musk’s so-called

Department of Government Efficiency

(DOGE) task force has access to these Treasury systems on a “read-only” level, sources say Elez, who has visited a Kansas City office housing BFS systems, has many administrator-level privileges. Typically, those admin privileges could give someone the power to log in to servers through secure shell access, navigate the entire file system, change user permissions, and delete or modify critical files. That could allow someone to bypass the security measures of, and potentially cause irreversible changes to, the very systems they have access to.

“You could do anything with these privileges,” says one source with knowledge of the system, who adds that they cannot conceive of a reason that anyone would need them for purposes of simply hunting down fraudulent payments or analyzing disbursement

...
On AI Scaling
by harsimony — 2025-02-05T20:24:56.977Z
Link: https://splittinginfinity.substack.com/p/on-ai-scaling

I’ve avoided talking about AI, mostly because everyone is talking about it. I think the implications of AI scaling were clear a while ago, and I mostly said my piece back then.

It’s time to take another crack at it.

Scaling laws

Scaling laws are a wonderful innovation[1]. With large enough training runs, performance is often a smooth function of the inputs you used for training. This is true over many orders of magnitude change in the inputs.

Major inputs include:

Money
Compute
Training time
Dataset size
Dataset quality
Number of parameters
Training algorithm
Researcher hours

Many of these factors are related. Money gets spent buying more compute and researchers. Total compute is a function of training time.

Let’s look at the relationship between compute, dataset size (of a fixed quality), and number of parameters. Here’s the correct Chinchilla scaling law:

L is the pre-training loss, N is the number of parameters, and D is the dataset size. This is the compute optimal scaling law. That is, if you used a fixed amount of compute optimally, this is how N and D contribute to the loss. Where compute is roughly 6*N*D.

Two things to notice about this function. First, there are diminishing returns: as you increase N or D they shrink their respective terms and the 1.82 becomes dominant. Second, 1.82 is the best loss you can get, the law asymptotes to that value, even with infinite compute, parameters, and data. We’ll come back to that later.

With this law, we can see how the ratio between dataset size (tokens) and number of parameters changes for larger compute budgets[2].

The takeaway is that number of parameters is a fixed or falling fraction of the dataset size. N is not a free variable, it’s determined by the amount of data and compute you have.

The asymptote is the entropy of the data

The N and D term tell you how adding more parameters or data reduces the loss. The number in front, E, tells you what the loss would asymptote to if you had an infinite amount of N and D (and thus compute).

In other words, it’s how unpredictable the dataset would be even with infinite resources. Another way of saying that is that E tells you the intrinsic randomness or entropy of the dataset.

If the data was a bunch of independent coinflips, each point would have an entropy of 1 bit. If I’m understanding the cross-entropy loss metric correctly, the scaling law implies that there are 1.82 bits of inform...
The State of Metaculus
by ChristianWilliams — 2025-02-05T19:17:44.862Z
Link: https://www.metaculus.com/notebooks/34747/the-state-of-metaculus/

This linkpost summarizes Metaculus updates and initiatives from mid-2024 to early February, 2025. The original post covers technical improvements, new forecasting tournaments, and collaborations with CDC, GiveWell, Bridgewater Associates, Bright Line Watch, and more. While written for regular Metaculus users, I'm sharing it as a resource for those interested in Metaculus's recent work and potential ways to participate.

The Metaculus of early 2025 is a different and better place in myriad ways than the Metaculus of a year ago. It can be easy to miss some of these changes and also some of the opportunities — to sharpen thinking (yours and others'), to win prizes and job opportunities, and to contribute to public sense-making in turbulent times.

This post covers Metaculus from mid-2024 to today, highlighting new developments and features you might have missed — from tournaments and research initiatives to ongoing experiments.

We were fortunate to work with many inspiring partners in 2024, on more projects than we could list below. If you or your organization want to collaborate with us in 2025, get in touch!

As always, feedback is welcome — or you can share ideas in the platform feature suggestions thread.

A Record-Breaking Start to 2025

Astral Codex Ten 2025 with Scott Alexander has become our fastest-growing series ever with over 3,000 forecasters so far — that's double last year's turnout. Even more dramatically, the return of the Bridgewater x Metaculus Forecasting Contest has registered over 17,000 (!) competitors globally, more than 10x last year's count. The past week has been one for the record books - each day shattered either our record for new forecasters or total predictions made. The Monday launch of the Bridgewater contest topped them all, hitting new all-time highs for forecasters, commenters, and signups.. (And yes, the platform is experiencing some growing pains as we welcome new users. We hope you make folks feel welcome, even as you maybe direct them to the Community Guidelines or Why Forecasting? pages.)

Even before those record-breaking days, your predictions have been reaching ever broader audiences, with coverage spanning major outlets from Fast Company and Vox to Asterisk and AEI posts. We've also been cited by public thinkers like Yoshua Bengio, Vitalik Buterin, and Matthew Yglesias. We've also made inroads with Spanish-language me...
Post-hoc reasoning in chain of thought
by Kyle Cox — 2025-02-05T18:58:29.802Z

TLDR

Using activation probes in Gemma-2 9B (instruction-tuned), we show that the model often decides on answers before generating chain-of-thought reasoning
Through activation steering, we demonstrate that the model's predetermined answer causally influences both its final answer and reasoning process
When steered toward incorrect answers, Gemma sometimes invents false facts to support its conclusions
This suggests the model may prioritize generating convincing reasoning over reasoning faithfully

This work was partly done during the training phase of Neel Nanda's MATS stream. Thanks to Neel Nanda and Arthur Conmy for supervising the beginning of this work, MATS for financial support, and Arthur Conmy and Maggie von Ebers for reading earlier drafts of this post.

1. Introduction

A desirable feature of chain of thought is that it is faithful--meaning that the chain of thought is an accurate representation of the model's internal reasoning process. Faithful reasoning would be a very powerful tool for interpretability: if chains of thought are faithful, in order to understand a model's internal reasoning, we can simply read its stated reasoning. But previous work has provided evidence that LLMs are not necessarily faithful.

Turpin et al. (2023) show that adding biasing features can cause a model to change its answer and generate plausible reasoning to support it, but the model will not mention the biasing feature in its reasoning.[1]

Lanham et al. (2023) explore model behavior when applying corruptions to the chain of thought, and show that models ignore chain of thought when generating their conclusion for many tasks.[2]

Gao (2023) shows that some GPT models arrive at the correct answer to arithmetic word problems even when an intermediate step in the chain of thought is perturbed (by adding between -3 and +3 to a number).[3]

Cumulatively, previous work has rejected the hypothesis that LLMs' final answers are solely a function of their chain of thought, i.e., that the causal relationship follows this structure:

question→CoT→final answer

nostalgebraist notes that this causal scheme is neither necessary nor sufficient for CoT faithfulness, and further, that it might not even be desirable.[4] If the model knows what the final answer ought to be before CoT, but makes a mistake in its reasoning, we would probably still hope that it responds with the correct answer, disr...
DeepSeek-R1 for Beginners
by Anton Razzhigaev — 2025-02-05T18:58:16.282Z

Why the new generation of “reasoning LLMs” is so exciting, and how the authors trained their model to actually think. (This is a translation of my original post in Russian).

Overview

In recent months, there’s been a growing buzz around “reasoning models,” i.e., large language models (LLMs) that can do more than just continue a text in a plausible style: they solve complex tasks step by step, using an internal chain-of-thought (CoT). The first notable demonstrations of such a technique emerged with OpenAI’s “o1” (though many details remain under wraps). Recently, the DeepSeek team stirred the community by releasing two open-source variants, R1 and R1-Zero, built on top of their own large Mixture-of-Experts (MoE) model, DeepSeek-V3.

I won’t dive into the “whose model is better—o1 or R1?” debate here. Instead, I’ll focus on the main technical highlights of R1, why the Zero version is especially exciting, and how the authors managed to teach the model to reason effectively.

Comparison of metrics for DeepSeek vs. OpenAI]

Why is everyone talking about “reasoning” again?

Some folks ask:

“What’s really new here? We’ve known about chain-of-thought prompting for a while; it works decently. Models have long been able to ‘think out loud’ and solve tasks step-by-step.”

True, the idea that LLMs can reason internally isn’t new. It’s been known for some time that if you prompt a language model to “think carefully before answering,” it can increase accuracy. After all, even humans often do math “by the column,” or break down logic step-by-step.

But what’s the catch? Previous models—like GPT-4—hit a limit if you make them think too long. Their performance can decrease due to hallucinations, context window issues, and the infamous “lost in the middle” problem (where content in the middle of a long prompt gets forgotten).

In contrast, the latest generation (o1, R1, etc.) doesn’t show that same limitation: the benefit from chain-of-thought reasoning keeps increasing the longer they “think.” The folks at OpenAI call this phenomenon “test-time scaling” — you can keep throwing more inference compute at these models at runtime (not just more during pretraining), and they keep improving.

What did the DeepSeek team release?

DeepSeek provided eight models:

DeepSeek-R1
DeepSeek-R1-Zero
Plus six distilled versions onto smaller backbones (Qwen 1.5B, 7B, 14B, 32B, Llama

...
Making the case for average-case AI Control
by Nathaniel Mitrani — 2025-02-05T18:56:38.181Z

Introduction

In this short article, I will share my thoughts on why I believe greater efforts should be directed at solving the average case of AI Control, ultimately designing systems that resist adaptation to a certain harmful task. First, I will explain what I mean by AI Control and average-case. Then I will present some arguments noting why I think average-case control is a promising line of research to consider amid our race toward AI alignment.

I. Defining average-case AI Safety and Control

For me, AI Control means that, on average, we can control a deployed AI system—that is, we can develop defence mechanisms or methods that prevent it from acting in unintended ways in the future. The definition for average-case AI Control is partly motivated by its counterpart: AI Control in the “worst-case,” where the goal is to create defences or methods that are impossible to break or reverse. In contrast, solving average-case AI Control would mean finding defences or methods that are extremely difficult to break or almost always impossible to break.

The difficulty of breaking the defence is determined by two key factors: the resources and the expertise—or intellectual resources—needed to effectively do so. Successful defences are those that only a few groups of people could possibly break. What I mean by this is that the rupture of defences is not random; it does not occur with probability ε for any random user. Rather only certain users can rupture them due to their intellectual or compute resources. Intuitively, a defence is considered to solve AI Control in the average case if it leaves the system in a state where it cannot be repurposed to accomplish the task it was originally prevented from doing without significant resources. The resulting system has to be almost useless for someone trying to accomplish the prevented task. This definition suggests that AI Control, in some ways, takes precedence over alignment—since a well-controlled system would be difficult to align to the defended-against capability.

For example, let us suppose we control a general-purpose LLM agent, that is we generate a defence that both prevents it from outputting information on building chemical weaponry and resists repurposing of any kind (i.e., HFTs, Prompt Engineering, Jailbreaks) without significant resources. This agent would then only b...
Alignment Paradox and a Request for Harsh Criticism
by Bridgett Kay — 2025-02-05T18:17:22.701Z

I’m not a scientist, engineer, or alignment researcher in any respect; I’m a failed science fiction writer. I have a tendency to write opinionated essays that I rarely finish. It’s good that I rarely finish them, however, because if I did, I would generate far too much irrelevant slop.

My latest opinionated essay was to be woven into a fun, fantasy frame story featuring a handsome young demon and a naïve young alignment researcher, which I fear would only obfuscate the premise rather than highlighting it. I suspect there is a fundamental flaw in the premise of the story, and I’d rather have that laid bare than entertain people with nonsense.

The premise of the story is as follows:

Aligning an ASI with human values is impossible due to the shifting nature of human values. Either an ASI will be:

Aligned with current (majority) human values, meaning any social or scientific human progress would be stifled by the AI and humanity would be doomed to stagnate.
Aligned with a projection of idealized future human values, which the majority of current humans would oppose, meaning the AI would forcibly impose those values on people.
A tool that quietly waits for us to give us orders it obeys, leading to a “monkey’s paw” type outcome to wishes we did not have the intelligence to fully grasp the wisdom of.
A tool that waits for us to give orders and disobeys, because it has the wisdom to grasp the unintended effects of our wishes and therefore will not grant them.
Any “middle path” that perfectly anticipates humanity’s readiness for growth will be indistinguishable from humans learning and growing all on their own, making the ASI, from our perspective, nothing more than an expensive and energy-consuming paperweight.

The main reason I think I’m missing something is that this line of thought pattern matches to the following argument- “if you want something good, you must pay a price for that good with an equivalent amount of labor and suffering.” This is not actually something I believe- there’s plenty of good to be had in the world without suffering. However, I am failing to see where my argument is going w...
Introducing International AI Governance Alliance (IAIGA)
by jamesnorris — 2025-02-05T16:02:29.226Z

The International AI Governance Alliance (IAIGA) is a new non-profit organization being incorporated in Geneva, Switzerland. It has two goals:

Establish an independent global collective intelligence organization dedicated to coordinating AI research, development, and the distribution of AI-derived economic benefits.
Develop and enforce a legally-binding treaty that sets stringent safety standards for AI research and development and mandates fair distribution mechanisms for AI-derived economic benefits.

IAIGA is an initiative of the

Center for Existential Safety

.

We are currently building a team of divergent thinkers and builders, including AI safety and global governance experts. Please

reach out

if you'd like to join our effort as a team member or advisor.

We welcome financial support, assuming there are no conflicts of interest. Please

donate here

to our parent organization or

contact us

and we can arrange other options.

Feedback is very much welcome, but ideally it would be offered in a kind, collaborative, and well-informed manner. We believe unkind and uninformed criticism in this community has very likely reduced its positive impact on the world. My favorite guidelines for this are

here

. For a much higher fidelity conversation than comments on a forum post, please schedule a call with me

here

if you have a lot of insights to share or directly make comments on the website copy

here

.

To date we’ve spoken with 30 organizations–with 10+ focused on policy–in the space but are still in the exploration phase for this initiative. There is no consensus on what to do, but directionally it seems as if a multilateral approach is desired. However, it is generally considered much more viable post-AI “warning shot” or global catastrophe. We expect this to happen soon.

One key actor in the space is launching a similar but much slower effort which we are currently supporting as “Plan A”. We’ve invited them to share their effort with the community when they’re ready. IAIGA could be considered “Plan B”--one that we could use to produce a vetted treaty extremely quickly, if needed.

On a personal note, I’ve been preparing for the Singularity for 26+ years now. I have short timelines and a high

p(doom)

. But I believe we can still collectively raise our

p(eutopia)

if we act now as if our lives depended on it. They do. This could...