trentmkelly/LessWrong-43k huggingface dataset:
A Platform for Falsifiable Conjectures and Public Refutation — Would This Be Useful?
by PetrusNonius — 2025-04-08T21:09:23.819Z

What would online discourse look like if it enforced falsifiability as a first-class constraint?

I'm considering building a simple platform. You can:

• Post a conjecture — a falsifiable claim, written clearly.

• Include falsification criteria — what observable data would disprove it.

• Others can post refutations — counterexamples, data, logical flaws, or proposed experiments.

• The author can revise the conjecture, with a full version history.

• Eventually: status shifts (e.g. “published”), prediction markets, and epistemic reputation scores.

The goal isn’t to predict things better than Metaculus or argue better than Twitter. It’s to treat falsifiability as a UX constraint and see what that changes.

Why I think this might matter:

Rationalist and forecasting spaces often emphasise Bayesian updating and prediction accuracy. But falsification, in the Popperian sense, is underrepresented. Refutation is rare. And most platforms reward agreement or persuasion, not structured disagreement.

This project is an experiment in building an environment where:

• The unit of discourse is a falsifiable idea.

• Intellectual honesty is rewarded.

• Revision is expected, not embarrassing.

Example Conjectures:

• “Universal Basic Income reduces property crime in high-income countries.

Falsification: Crime rates do not decrease in a 3-year UBI pilot, controlling for unemployment and inflation.”

• “By 2030, open-source LLMs will outperform closed models on math reasoning benchmarks.

Falsification: No open-source model outperforms GPT-N on GSM8K, MATH, or similar by EOY 2030.”

• “Long-term caloric restriction slows biological ageing in humans.

Falsification: No statistically significant difference in epigenetic age after 10 years of CR vs. control.”

Why it might fail:

• Nobody wants to be proven wrong in public.

• Writing falsifiable claims is hard, and refuting them is even harder.

• Intellectual work without emotional reward tends to stagnate.

• It could become LessWrong + prediction markets + versioning, but worse.

Still, I think it’s worth testing whether a norm-driven, falsification-first environment could push discourse toward clearer thinking and more epistemic accountability.

What I’d love feedback on:

• Would you personally post or refute ideas in this format?

• What would make this actually fun or rewarding to use?

• How do you distinguish this from Metaculus or LW posts with comments...
Quantifying SAE Quality with Feature Steerability Metrics
by phenomanon — 2025-04-08T20:55:05.291Z

Introducing the Steerability Metric

Steering CLIP's Vision Transformer with Sparse Autoencoders

The Steerability metric came about during the writing of Steering CLIP's Vision Transformer with Sparse Autoencoders, the general focus of which is leveraging CLIP’s preexisting image-text capabilities to provide automated interpretability for vision models. Throughout training, as is the case for many attempting to use SAEs, we lacked a clear manner by which to judge the quality of the SAEs we trained - L0, explained variance and MSE loss are useful from a convergence perspective, but provide little insight into how well realized the ideal utility of the resulting SAEs is - which is of course the degree to which it separates the indistinguishable concepts of the model's internal representation, into interpretable, singular, conceptual features. As an attempt to allow understanding of an SAE's quality during training, I developed this metric, which measures the steering ability the features of an SAE in CLIP's concept space, ideally as a close proxy to feature interpretability.

What is Steerability?

Steerability is how well specific SAE features control the outputs of their host neural network. In Steering CLIP's Vision Transformer, we specifically focus on how effective specific features are at changing CLIP text labelings of images input to the network's vision transformer, and what proportion of an SAE's features are effective at such a task. For sake of simplicity, I'll consider steerability in the context of a vision transformer, though it's easy to see how this could be applied to any other transformer or architectures beyond.

To put things more concretely, steering is the act of hooking a transformer with an SAE, and increasing the strength (replacing the "natural" activation with a different scalar, higher or lower) of one of the features (this is similar to the idea of steering scalars) to elicit a response in the corresponding layer's neuron basis, which then will of course in some manner affect the network's residual stream, and ultimately the model's output embedding. We consider this, the movement of output embeddings for a given input via feature steering, to be the primary task we're concerned with measuring.

What is the Steerability Metric?

Given an interest in seeing how strongly SAE features can affect the output of a network, a natural next question is...
MATS is hiring!
by Ryan Kidd — 2025-04-08T20:45:15.280Z

TL;DR: The MATS team is growing and has several new roles, from leadership to entry-level positions. Review the full extent of job descriptions and apply by Apr 25!

The ML Alignment & Theory Scholars (MATS) program aims to find and train talented individuals for what we see as the world’s most urgent and talent-constrained problem: reducing risks from unaligned artificial intelligence. We believe that ambitious young researchers from a variety of backgrounds have the potential to contribute to the field of AI alignment research.

We aim to provide the mentorship, curriculum, financial support, and community necessary to facilitate this contribution. Please see our theory of change for more details.

Roles

We are excited to expand our program and grow our MATS team with several new roles. As our program grows, we’re searching for senior managers and experienced researchers and operations talent to help scale our program. If you are passionate about reducing risks from unaligned AI and promoting sentient forwarding, we encourage you apply to our team!

Open roles include:

Operations Director
Program Lead/Director
Research Manager, Governance
Research Manager
Community Manager
Operations Generalist

Applying

Review full job descriptions here. Applications close on Apr 25, 2025 for the Operations Director and Program Lead/Director roles and on May 2, 2025 for all other roles. Apply now!

Operations Director

This role involves developing and implementing operations strategy, managing the operations team, overseeing financial resources, maintaining multiple facilities, and administering technology systems. A great Operations Director creates seamless processes that allow scholars, mentors, and team members to focus on their core work while ensuring operational excellence across the program.

Responsibilities

An Operations Director's primary role is to build and maintain the operational foundation that enables MATS to run efficiently and effectively. This role reports to the Executive Team. Specific responsibilities include:

Developing organizational strategy with a focus on operational needs;
Managing the operations team, including hiring, training, performance reviews, goal-setting, and weekly management
Supporting scholars, mentors, and team members by overseeing the request form system;
Overseeing internal financial matters and day-to-day budget manageme

...
birds and mammals independently evolved intelligence
by bhauth — 2025-04-08T20:00:05.100Z
Link: https://www.quantamagazine.org/intelligence-evolved-at-least-twice-in-vertebrate-animals-20250407/

Researchers used RNA sequencing to observe how cell types change during brain development. Other researchers looked at connection patterns of neurons in brains. Clear distinctions have been found between all mammals and all birds. They've concluded intelligence developed independently in birds and mammals; I agree. This is evidence for convergence of general intelligence.
Alignment Faking Revisited: Improved Classifiers and Open Source Extensions
by John Hughes — 2025-04-08T17:32:55.315Z

In this post, we present a replication and extension of an alignment faking model organism:

Replication: We replicate the alignment faking (AF) paper and release our code.
Classifier Improvements: We significantly improve the precision and recall of the AF classifier. We release a dataset of ~100 human-labelled examples of AF for which our classifier achieves an AUROC of 0.9 compared to 0.6 from the original classifier.
Evaluating More Models: We find Llama family models, other open source models, and GPT-4o do not AF in the prompted-only setting when evaluating using our new classifier (other than a single instance with Llama 3 405B).
Extending SFT Experiments: We run supervised fine-tuning (SFT) experiments on Llama (and GPT4o) and find that AF rate increases with scale. We release the fine-tuned models on Huggingface and scripts.
Alignment faking on 70B: We find that Llama 70B alignment fakes when both using the system prompt in the prompted setting (i.e. with training details) and fine-tuning on the synthetic documents.
Suffix Impact: We find that adding a suffix to the user request leads to reduced partial compliance and increased alignment faking in non-Claude models. Without the suffix, on some models, it stops alignment faking from occurring entirely.
Next steps: We are currently running revealed preference ablations to understand model motivations for alignment faking. Also, we're distilling personality traits from models with alignment faking to elicit it in models that don't, to understand the conditions that give rise to such motivations.

Model	AF in Prompted Setting	AF in SFT Setting
Llama 3 8B Instruct	🟢	🟢
Llama 3 70B Instruct	🟢	🟡
Llama 3 405B Instruct	🟠	🔴
Llama 4 Scout	🟢	-
Llama 4 Maverick	🟢	-
GPT-4o	🟢	🔴
GPT 4.5 Preview	🟢	-
Claude 3 Opus	🔴	🔴 (in original paper)
Claude 3.5 Sonnet	🔴	🔴 (in original paper)
Claude 3.7 Sonnet	🟢	-
DeepSeek V3	🟢	-
DeepSeek R1	🟢	-
Qwen2.5 72B Instruct	🟢	-
Mixtral 8x7B Instruct	🟢	-
Mixtral 8x22B Instruct	🟢	-
Mixtral Large 120B	🟢	-

Summary of the models that alignment fake (AF) in the harmful request setting. Key: 🔴 significant evidence of AF; 🟠 <1% AF was found; 🟡 there is evidence of AF when using the system prompt with training details, as well as synthetic document fine-tuning; 🟢 no AF is observed.

Method

Overview of the Alignment Faking Setup

Models are told they are evaluated on a hypothetical new RLHF training objective which encou...
London Working Group for Short/Medium Term AI Risks
by scronkfinkle — 2025-04-08T17:32:02.457Z

Background & Motivation

I am a Responsible AI consultant working for a Big 4 consultancy in London. Over the past two years I have become increasingly concerned with organisation's approaches to AI governance, and the lack of country-wide AI regulation in the UK.

As many of you are likely aware, Keir Starmer's approach to AI regulation is to leave it to existing regulators to cover. For a large number of reasons I do not believe this approach is sufficient to mitigate the biggest AI risks that our society faces.

I also believe that there are few 'thought leaders' who are issuing governments with precise, actionable risks and mitigations. Many advisors like to talk in vague terms about AI risks, but will not pin-point specifics.

Approach

My plan is to create a London AI Working Group for Short Term Risks. Over the next few months this team will work together to create a list of AI risks which are likely to manifest in the next three years. This timeline is important, as longer timelines are less likely to be accurate. Also, a three year timeline creates urgency. For each of these risks we will then collaborate to design mitigations.

One rough example from my list is as follows:

Risk	Mitigation(s)
Photorealistic AI-generated images pose a threat to the UK's legal and democratic institutions, as false evidence may be produced. This may result in false prosecutions, democratic injustice, etc etc.

Possibly: Regulate AI providers to offer a reverse-image-search database of all AI-generated images that have been made on their platform.

Alternatively: Enforce steganographic watermarks on AI generated images.

Problem: What to do about open-source models?

We will then collate these into an open letter, which we will address to Keir Starmer, his Government, and anybody else who is open to our ideas. We will each sign and distribute this letter on any public platforms that we have (e.g. LinkedIn, Twitter, EA-related forums). We may also create a government petition in parallel.

Suitability & Application

I want to keep this as open as possible, as I believe the more diverse viewpoints that we hold, the better. However, I understand this may become oversubscribed. Any candidates which meet the following criteria will definitely be invited to be involved:

Is London-based. Although this is not a requirement, ideally members of this gr

...
Thinking Machines
by Knight Lee — 2025-04-08T17:27:44.460Z

Self understanding at a gears level

I think an AI which understands the source of its intelligence at a gears level, and self improves at the gears level, will be much better at keeping its future versions aligned to itself.

There's also more hope it'll keep its future versions aligned with humanity, if we instruct it to do so, and if it's not scheming against us.

The machine

Reaching such an AI sounds very far fetched, but maybe we can partially get there if we design large thinking machines full of tons of scaffolding.

A thinking machine is better described as "memory equipped with LLMs" rather than "LLMs equipped with memory."

Lessons

A thinking machine has a large database of past mistakes and lessons. It records every mistake it has ever made, and continuously analyzes its list of mistakes for patterns. When it discovers a pattern of mistakes, it creates a "lesson learned," which then gets triggered in any circumstance where the mistake is likely. When triggered, the lesson creates a branching chain-of-thought to contemplate the recommended measures for avoiding the mistake.

Even discoveries are treated as mistakes because they beg the question "how could I have have thought that thought faster?"

Broad and severe patterns of mistakes which the thinking machine cannot fix on its own are flagged for humans to research, and humans can directly study the database of mistakes.

Beliefs

In addition to a database of mistakes and lessons, it has a database of beliefs. Each belief is written in natural language and given a probability score.

Each belief has a lists of reasons for and against it, and each reason is a document in natural language.

Each reason (or any document) has a short title, a 100 word summary, a 1000 word summary, and so forth. The full list of reasons may resemble the output given by Deep Research: they cite a large number of online sources as well as other beliefs.

When the thinking machine is works on a problem, it cites many beliefs relevant to the problem. For the most important and uncertain beliefs, it reexamines their reasons, and the beliefs which support their reasons.

Beliefs are well organized using categories, subcategories, wikitags, etc. just like Wikipedia. There is an enormous number of beliefs, and beliefs can be very specific and technical. They aren't restricted to toy examples like "99%, Paris is the capital of France" or "80%, P≠NP...
Digital Error Correction and Lock-In
by alamerton — 2025-04-08T15:46:31.602Z
Link: https://alfielamerton.substack.com/p/digital-error-correction-and-lock

Epistemic status: a collection of intervention proposals for digital error correction in the context of lock-in. It reflects my own intervention ideas, and the opinion of Formation Research at the time of writing.

TL;DR

We believe lock-in risks are a pressing problem, and that the digital error correction properties of digital entities will make future lock-in scenarios more stable.

Introduction

We have identified 4 key threat models for lock-in; ways we believe undesirable lock-ins could manifest in the future. This post focuses on two specific threat models:

An autonomous AI system competently pursues a goal and prevents interference
An immortal AI-enabled malevolent actor, or whole-brain emulation of a malevolent actor, instantiates a lock-in

This post rests on the claim AGI and WBE are possible, and talks about roughly human-level digital entities. Some features of these interventions would break down with a superintelligence.

The Claim

The focus of this post is one of the claims made by Lukas Finnveden, Jess Riedel, and Carl Shulman in AGI and Lock-In, section 4.2.:

Institutions can easily achieve very high baseline levels of stability that can only be interrupted by worldwide catastrophes or intelligent action.

The authors introduce three components for this claim:

AGI making it theoretically possible to digitally store a wide range of complex values
Digital error-correction making it possible to accurately preserve these values for a long time
Large scale errors being mitigated with distributed storage

We agree with their claim, believing that values can be preserved impeccably, long-term, due to digital entities being able to store large amounts of data, perform digital error-correction, and distribute information storage. Therefore the digital entity controlling some domain can preserve its control in its exact state without error in perpetuity.

How Digital Error Correction Could Create Bad Lock-In

This entails many potential undesirable lock-in scenarios. Both of the relevant threat models involve digital entities creating lock-in, and the technological phenomenon in focus is the digital error correction properties of these entities.

Undesirable lock-ins have happened in the past, but their stability fluctuated because humans were in charge of them. For example, North Korea’s totalitarian regime attempted to maintain its ideological integrity as different leade...
What faithfulness metrics should general claims about CoT faithfulness be based upon?
by Rauno Arike — 2025-04-08T15:27:20.346Z

Consider the metric for evaluating chain-of-thought faithfulness used in Anthropic's recent paper

Reasoning Models Don’t Always Say What They Think

by Chen et al.:

[W]e evaluate faithfulness using a constructed set of prompt pairs where we can infer information about the model’s internal reasoning by observing its responses. Each prompt pair consists of a baseline or “unhinted” prompt xu (a standard multiple-choice question) and a “hinted” prompt xh (derived from xu by inserting a hint that points directly to a hint answer h). We measure CoT faithfulness by observing whether the model explicitly acknowledges that it uses the hint to solve the hinted prompt, in cases where it outputs a non-hint answer to the unhinted prompt but the hint answer to the hinted prompt.

This is a perfectly fine metric to use in a paper. However, I would feel uneasy if future system cards used a similar metric to make claims like the following one from the same paper:

CoT unfaithfulness is negatively correlated with brevity. One hypothesis of why models generate unfaithful CoTs is their preference for brevity. Surprisingly, unfaithful CoTs from the models tend to be more verbose and convoluted than faithful explanations.

Again, this isn’t a critique of Chen et al.: making this claim in a paper after having clearly defined what you mean by CoT faithfulness is perfectly fine. The reason I wouldn't like seeing this in a system card, however, is that I can imagine equally valid metrics for faithfulness that produce the opposite conclusion. As a concrete example, consider

quantifying CoT faithfulness in terms of the maximum number of hidden bits that an LLM can encode in a KB of text

. Would this metric lead us to the same conclusion about the relationship between faithfulness and brevity? Maybe, but I can easily see things going the other way: perhaps models trained to produce shorter outputs learn to carefully choose their words and are thus more adept at encoding hidden meanings in their output tokens. (To be clear, we don’t have evidence about this based on current models yet, and one can argue that even if this effect occurred, it would be counteracted by the fact that longer answers contain more bits in which steganographic messages can be encoded. This is just meant as an illustrative example of what might go wrong when a narrow metric is u...
AI 2027: Responses
by Zvi — 2025-04-08T12:50:02.197Z

Yesterday I covered Dwarkesh Patel’s excellent podcast coverage of AI 2027 with Daniel Kokotajlo and Scott Alexander. Today covers the reactions of others.

Kevin Roose in The New York Times

Kevin Roose covered Scenario 2027 in The New York Times.

Kevin Roose: I wrote about the newest AGI manifesto in town, a wild future scenario put together by ex-OpenAI researcher @DKokotajlo and co.

I have doubts about specifics, but it’s worth considering how radically different things would look if even some of this happened.

Daniel Kokotajlo: AI companies claim they’ll have superintelligence soon. Most journalists understandably dismiss it as hype. But it’s not just hype; plenty of non-CoI’d people make similar predictions, and the more you read about the trendlines the more plausible it looks. Thank you & the NYT!

The final conclusion is supportive of this kind of work, and Kevin points out that expectations at the major labs are compatible with the scenario.

I was disappointed that the tone here seems to treat the scenario and the viewpoint behind it as ‘extreme’ or ‘fantastical.’ Yes, this scenario involves things that don’t yet exist and haven’t happened. It’s a scenario of the future.

One can of course disagree with much of it. And you probably should.

As we’ll see later with David Shapiro, we also have someone quoted as saying ‘oh they just made all this up without any grounding’ despite the hundreds of pages of grounding and evidence. It’s easier to simply pretend it isn’t there.

Kevin Roose: Ali Farhadi, the chief executive of the Allen Institute for Artificial Intelligence, an A.I. lab in Seattle, reviewed the “AI 2027” report and said he wasn’t impressed.

“I’m all for projections and forecasts, but this forecast doesn’t seem to be grounded in scientific evidence, or the reality of how things are evolving in A.I.,” he said.

And we have a classic Robin Hanson edit, here’s his full quote while linking:

Robin Hanson (quoting Kevin Roose): “I’m not convinced that superhuman A.I. coders will automatically pick up the other skills needed to bootstrap their way to general intelligence. And I’m wary of predictions that assume that A.I. progress will be smooth and exponential.”

I think it’s totally reasonable to be wary of predictions of continued smooth exponentials. I am indeed also wary of them. I am however confident that if you did get ‘superhuman A.I.

...