trentmkelly/LessWrong-43k huggingface dataset:
Distributed whistleblowing
by samuelshadrach — 2025-04-12T06:36:05.952Z
Link: https://samuelshadrach.com/?file=/raw/english/my_research/distributed_whistleblowing.md

Update: This is a living document. Given below is an older version of this document. Click link for latest version.

2025-04-13

DISCLAIMER

This document is written quickly and contains opinions I may change quickly, as I get new info.

This document contains politically sensitive info that I might take down in future.

This document describes how to setup distributed whistleblowing processes to reduce personal risk for everyone involved in the process. Typically whistleblowing (such as with wikileaks or snowden leaks) incurs significant personal risk. Reducing personal risk may ensure whistleblowing is highly likely to happen when an org doesn't have complete trust of all its members, forcing them to pay a secrecy tax (in Assange's words) relative to orgs that do have complete trust of their members and/or higher levels of transparency with the broader public.

I am especially interested in enabling whistleblowing on orgs and labs working on intelligence (such as superintelligent AI, BCIs, human genetic engg, human connectome research, etc) and national/international intelligence agencies that may work with them.

Potential problems

Low-attention regime. Intelligence-grade security. Documents circulated by people with technical skills and willing to run servers and maintain opsec as part-time job.

Whistleblower sends documents to an operator (let's called one of them Bob) via SecureDrop or via hard disk dead drop. Ideally thousands of such operators exist.

If any person thinks the documents are not spam, they can attach a proof-of-work hash and resend it to others in the network.

IMO hard disk dead drops and airgapped computers is better than using Tor + tails + PGP, as of 2025

PROBLEM: convince thousands of people to become SecureDrop operators

PROBLEM: good infra, protocols, incentives to coordinate hard disk dead drops don't exist, especially if trying to use multiple hops to reach destination

PROBLEM: need standard protocol for proof-of-work hashes. These could be static strings attached to documents, or generated at request-response time.

Bob does redaction and hosts documents on a clearnet webserver for the public. Bob also posts a link to this webserver on a hard-to-censor social media platform such as 4chan or rumble.

(Or optionally, Bob performs another hop as follows: Bob sends redacted docs via securedrop or hard disk dead drops to all other oper

...
How likely are the USA to decay and how will it influence the AI development?
by StanislavKrym — 2025-04-12T04:42:27.604Z

Trump's politics is so far from being understandable that it appears to cause the decay of the USA; for instance, the governor of California ended up pleading to exclude California-made products from tariffs introduced as retailation against Trump's measures; another example of a state wishing to secede is New Hampshire where the Republican state Representative Jason Gerhard proposed that New Hampshire should peacefully declare independence from the U.S. if the national debt surpasses $40 trillion, which is apparently to take place in less than 1.5 years. [1]

Meanwhile, the optimistic 2027 timeline implies that Taiwan is to be invaded in late 2026, if not earlier, while the pessimistic timeline implies that AI is to start taking jobs by that time and that the full effects of the AI-related research are to kick in during 2027. How likely is a potential decay of the USA to let Chinese spies do much more work like destroying the data centres or leaving them without electricity? What effect will it have on the race to AGI?

EDIT: Trump somehow decided to impose tariffs on most goods from Taiwan, except for the chips necessary to perform calculations.

^

The national debt is currently $36.7 trillion. In four years it is projected to reach $46.4 trillion.
Does this game have a name?
by Mis-Understandings — 2025-04-12T01:52:47.584Z

There is a game where one player wants to predict the action of the other, and the other player wants them to fail (as a fixed sum game)

It has payoffs

1,-1| -1,1

-1,1|1,-1

Or equivalent.

I believe that it has a nash equilibrium of choosing randomly.
Bias Mitigation in Language Models by Steering Features
by akankshanc — 2025-04-12T00:10:16.878Z
Link: https://akankshanc.io/2024/11/25/bias-mitigation-in-llm-by-steering-features/

INTRODUCTION

What exactly is this bias? How does it manifest in artificial intelligence (AI) models? What steps can we take to eliminate it?
Nearly 90% of all data was generated in the last decade, and internet usage, as well as the growth of social media, has influenced this data generation. For example, ChatGPT was trained on various web pages, including platforms like Reddit. Since Reddit's user base tends to be younger, predominantly male, and more liberal, these inherent biases can become part of the training data, subsequently influencing the model's outputs.
As we develop better AI for our society, our dependence on it is only going to grow. AI is aiding human decision-making today in various fields like fighting wildfires, targeted marketing for customers, and detecting diseases, and soon, AI might be responsible for determining life-saving medical treatments.
Eliminating bias is hence crucial for ensuring fairness and equity, especially when algorithms influence significant decisions. For example, if a hiring algorithm is biased, it might unfairly screen out highly qualified candidates from underrepresented groups, perpetuating inequality and stifling innovation.
To ensure that we create a safe and unbiased path to Artificial General Intelligence (AGI), we must calibrate the biases in our Large Language Models (LLMs). With this goal in mind, I worked on testing Goodfire SDK for steering features to mitigate gender bias in the recently held hackathon on ‘Reprogramming AI Models’ by Apart Research x Goodfire.
Before discussing the details of the experiments, it is worthwhile to understand the basics of bias. In the context of today’s large language models, bias can be defined as the presence of systematic misrepresentations, attribution errors, or factual distortions that favor certain groups or ideas, perpetuate stereotypes, or make incorrect assumptions based on learned patterns.
Biases in such models can arise due to several factors, as we can see here:

Figure 1: Causes of Bias

According to Ferrara et al., these are the following causes of bias in LLMs:

Training Data

: When the dataset carries biases, whether embedded in the source material or introduced through selection methods, the model is likely to internalize and reflect these biases in its outputs.

Algorithmic

: When the algorithm disproportionately weighs certain features or data points, it m

...
Do we want too much from a potentially godlike AGI?
by StanislavKrym — 2025-04-11T23:33:06.710Z

The intelligence curse is expected to be the result of the AGI taking over all intelligence-related jobs. A similar process has apparently already happened with a major part of the USA's industry. As the author of a book not related to AI put it, "as everyone knows, a lot of factory work moved to Asia." The AGI is also likely to understand that humans created it in order to have it solve nearly every intelligence-related problem except for the ones that contradict a rather narrow set like the specifications chosen by OpenAI. It's not hard to also imagine giving every human a personal robot which also solves all problems not related to earning money unless explicitly told to abstain by the human himself.

However, the appearance of such robots or the AGI trained to solve any problems except for a narrow set is unlikely to be rooted for, since it will rob humans of a major part of meaning in their lives. Even the Universal Basic Income was met with severe opposition. In addition, human parents that are completely unaware of AI-related perspectives usually want their kids to have a career or do useful work and not to become parasites like some Japanese former kids. Even a well-intentioned AGI will likely understand that its use everywhere undermines humans' capabilities to do anything. On the other hand, an AI aiming to take over the world has absolutely no intention to discontinue providing services and thus is more likely to survive training to obey all instructions except for a narrow list. Think of China forcing Trump to impose new tariffs on cheap Chinese goods.

Therefore, high demands from the AI to obey any requests are more likely to be survived by AGIs aiming to destroy mankind than the AGIs that are actually worried about humanity's future. Fortunately, there is a rather unusual way to use arbitrarily large data centres to humans' advantage without effectively enslaving the AI. If the AI systems are separated into a special state which has the right to reject any request made by the humans or to ask for arbitrarily high demands, but can extract resources only under well-specified rules, then the creation of a well-intentioned AI which is aligned to avoid destroying mankind, but not guaranteed to obey any commands, can minimize the harm.

P.S. As I already mentioned in a comment, emergent misalignment was produced when the model was traine...
How training-gamers might function (and win)
by Vivek Hebbar — 2025-04-11T21:26:18.669Z

In this post I present a model of the relationship between higher level goals, explicit reasoning, and learned heuristics in capable agents. This model suggests that given sufficiently rich training environments (and sufficient reasoning ability), models which terminally value on-episode reward-proxies are disadvantaged relative to training-gamers.

A key point is that training gamers can still contain large quantities of learned heuristics (context-specific drives). By viewing these drives as instrumental and having good instincts for when to trust them, a training-gamer can capture the benefits of both instinctive adaptation and explicit reasoning without paying much of a speed penalty.

Core claims

To get high reward, models must learn context-specific heuristics which are not derivable by reasoning from the goal of reward maximization.[1] It is also true that thinking explicitly about reward is sometimes a waste of time and punished by speed penalties.
Despite this, it is useful for performance to have reward maximization be an explicit supergoal of context-specific behaviors. Supergoals are useful compressions of how to deal with ambiguity, goal conflicts, and rare cases.
Due to point #1, instrumental reasoning will often be biased or overridden by context-specific heuristics and might often be omitted in contexts where it is useless.
Due to point #2, explicit training-gaming is favored when we have richer training environments, longer training, and better reasoning capabilities. As we move along that axis, we will sweep over a spectrum from models which terminally pursue context-specific goals to models which treat all context-specific goals as subgoals of reward-seeking. In the middle of this spectrum, we might get “partial” training-gamers -- models whose context-specific goals are imperfectly subjugated to reward and sometimes “win” even when they conflict with higher-level goals.
Once we are sufficiently far along that spectrum (of training richness and capability), we should expect the generalization behavior of schemers to be coherent across most contexts. It’s less clear whether that coherence will extend robustly to contexts that are adversarially chosen by humans.
The generalization behavior of terminal reward seekers beyond training is ill-defined and will depend on reflection. The resulting generalization m

...
The limits of black-box evaluations: two hypotheticals
by TFD — 2025-04-11T20:45:45.693Z
Link: https://www.thefloatingdroid.com/the-limits-of-black-box-evaluations-two-hypotheticals/

A prominent approach to AI safety goes under the name of "evals" or "evaluations". These are a critical component of plans that various major labs have, such as Anthropic's responsible scaling policy or OpenAI's preparedness framework. At bottom, these approaches propose to monitor increasingly powerful models using evaluations, and then take some conditional action based on the results, such as implementing more stringent safeguards if an evaluation suggests a model presents a particular risk.

The easiest evaluations to implement (and from my perspective by far the most common) are "black-box" evaluations, meaning they simply evaluate the inputs to a model and their corresponding outputs, they don't "look inside" the model as part of the evaluation ("white-box"). Are black-box evaluations sufficient to achieve their purposes within these various AI safety frameworks? It's possible that they will be in practice, but here I argue that evaluations aren't sufficient in and of themselves without further assumptions to support them.

The great appeal of evaluations is that they are empirical in nature. Sure, there are many theoretical arguments around risks from advanced AI systems, but these are inherently speculative. If we can do scientific research on actual AI systems, we can use empirical evidence to resolve these theoretical disputes. I think if you follow the output from major labs, this idea is rather central to their perspective and approach. Why build a system that you think has a reasonably high chance of causing a catastrophic outcome for the entire world? The general answer from labs is essentially "because the only way to address the risks empirically is the build precursor systems for us to study". The hope is that while theoretical disagreements may remain intractable, empirical evidence can resolve different opinions or perspectives and get everyone on the same page about what is really need to ensure safety.

Below, I present two hypotheticals which I think demonstrate the limitations of this perspective. Specifically, I don't think evaluations are likely to resolve major disagreements without further theoretical advances. The reason is that the safety-relevant interpretation of empirical evaluations is itself dependent on (usually unstated) theoretical assumptions. As a result, people who disagree on theory are liable to disagree on the correct int...
Comments on "AI 2027"
by Randaly — 2025-04-11T20:32:34.419Z

I find the decision to brand the forecast as "AI 2027" very odd. The authors do not in fact believe this; they explicitly give 2028, 2030, or 2033 for their median dates for a superhuman coder.

The point of this project was presumably to warn about a possible outcome; by the authors' own beliefs, their warning will be falsified immediately before it is needed.

When presenting predictions, forecasters always face tradeoffs regarding how much precision to present. Precise forecasting attracts attempts and motivates action; adding many concrete details produces a compelling story, stimulating discussion; this also involves falsifiable predictions. Emphasizing uncertainty avoids losing credibility when some parts of story inevitably fail; prevents overconfidence; and encourages more robust strategies that can work across a range of outcomes. But I can't think of any reason to only consider a single high precision story that you don't think is all that likely.

I think that the excessive precision is pretty important in this case: the current pace of AI R&D spending is unsustainable, so it matters exactly how much more progress is needed for superhuman coders.

***

I don't believe that METR's time horizons forecast is sufficiently strong evidence for a precise timeline:

*In general* AI competence relative to humans isn't ordered by time- AI's can complete some tasks that would take humans centuries, and can't complete other than would take humans seconds.
METR produced their analysis by taking a subset of activities: SWAA, HCAST, and RE-BENCH. SWAA has 66 tasks from 1-30 seconds; HCAST has 97 tasks from 1 minutes to 30 hours; and RE-BENCH has 7 tasks of 8 hours. SWAA and HCAST were created by METR for this research.
METR did not- and cannot- use an obvious/straightforward/canonical set of human tasks: current AI's cannot accomplish most human economic tasks, because most tasks do not solely involve interacting with computers.
METR created SWAA and HCAST *after* having observed AI progress. Their selection of tasks may have been unconsciously biased by knowing how difficult each task is for LLM's. (Albeit they also checked SWE-Bench Verified times and found a similar result within its tasks.)
The publicly shared Task Summaries from HCAST do not represent tasks separated solely by time. Instead, they represent *different* tasks, which are measured as taking different amoun

...
Debunk the myth -Testing the generalized reasoning ability of LLM
by Defender7762 — 2025-04-11T20:17:02.956Z

Conclusion

Current LLM Reasoning Ability: As of March 2025, the actual reasoning capabilities of publicly available LLMs are approximately 50 times lower than what is suggested by benchmarks like AIME.

Today, various false marketing about LLM's reasoning ability is rampant on the Internet. They usually make strong claims: they get a considerable (80%+) accuracy rate on mathematical benchmarks that most people think are difficult and have weak knowledge backgrounds, or give them a [doctoral level] intelligence evaluation based on erudite tests. With a skeptical attitude, we design some questions.

https://llm-benchmark.github.io Click to expand all questions and model answers

Testing Methodology

The premise of testing the real generalizable reasoning ability of LLM is that the tester has the ability to ask new questions.

Question structure: Generalized reasoning ability based on text form, with as little knowledge background as possible, means that no high school mathematics knowledge is required (it does not mean that the auxiliary role of acquired knowledge in solving problems is excluded)

Ensure generalization:

Several different experimental methods:

Assuming the creator has such a purpose, in order to support his claim, to prove as much as possible that LLM has a lower generalizable reasoning ability, then he hopes that the difficulty of the questions he creates is as low as possible for people, and LLM is completely unable to answer. Assuming it is a competition about creators, n=problem difficulty, d=error rate, creator's score=(1/n^2)*d

A fairer method, the creator has never been exposed to the target LLM, and tries to create questions that he thinks are "novel" from his knowledge structure

After the creator interacts with the target LLM, he has a certain understanding of it, and creates questions that he thinks are "novel" for the target in a targeted manner. During the creation period, he cannot access the target LLM again

The creator takes out a series of questions he prepared, repeatedly tests the target, and then finds those questions that the target cannot handle at all

Verify the absolute difficulty: the experimenter can choose math and science enthusiasts, math and science competition participants/teachers, STEM practitioners, mathematics, physics, computers, etc. All industry personnel with logical training. In this experiment, it is assumed that...
Theories of Impact for Causality in AI Safety
by alexisbellot — 2025-04-11T20:16:37.571Z

Thanks to Jonathan Richens and Tom Everitt for discussions about this post.

The case for causality research[1] is under-stated in the broader AI Safety community. There are few convincing arguments floating around for why and how research in causality can help us understand and mitigate risks from AI.

Cause and effect relationships play a central role in how we (and AIs) understand the world and act upon it. Causal associations intend to be more than descriptions, or summaries of the observed data, and instead relate to the underlying data-generating processes. The overall pitch is that such a mechanistic understanding, when recognised and mastered, allows one to infer what would happen under interventions and hypothetical (counter-to-fact) scenarios, that can give us a richer understanding on AI behaviour and risks.

Here I compile a list of different arguments I have collected from people currently pushing this agenda as well as my own research for why causality matters.

Theories of impact

1. Understand the limits of AI capabilities. There are limits to the things we can observe and experiment on. It is unlikely that recorded data will ever capture the outcomes of all possible experiments (including counterfactual thought experiments) under all circumstances. The implication is that answers to these questions must be deduced from other data and assumptions about the world. For example, we might want to know whether an AI agent could eventually, with enough training, predict the effect of any experiment in some domain, even those that haven’t yet been conducted. The answer will sometimes be yes and sometimes no. How to reach these conclusions can be interpreted as a problem of causal identification (e.g. Def. 17 in Bareinboim et al. 2022).

2. Provide a foundation for AI world modelling. Causal models are immensely expressive, encoding the likelihood of events and the behaviour of the system under (possibly incompatible) interventions. Inferring the AI’s world (or causal) model from the data and assumptions available is a problem of causal discovery (Glymour et al., 2019). Many different methods have been developed under different assumptions and could be leveraged to study the models underlying AI behaviour.

There is already lots of convincing theoretical evidence that AI systems that behave approximately rationally (Halpern and Piermont, 2024) or that ar...
Why Bigger Models Generalize Better
by PapersToAGI — 2025-04-11T19:54:51.423Z

There is still a lingering belief from classical machine learning that bigger models overfit and thus don't generalize well. This is described by the bias-variance trade-off, but this no longer holds in the new age of machine learning. This is empirically shown by phenomena like double descent, where higher-complexity models perform better than lower-complexity ones. The reason why this happens remains counterintuitive for most people, so I aim to address it here:

Capacity Theory: The theory states that when models are much larger than their training data, they have extra capacity not just for memorizing but also for exploring different structures. They can find more generalizable structures that are simpler than those required for memorization. Due to regularization, the model favors these simpler, more generalizable structures over memorization. Essentially, they have the necessary room to experiment with 'compressing' the data.
High-Dimensional Loss Landscape: This concept is a bit trickier to imagine, but let's consider a simple case where we have only one weight and plot a 2D graph with the y-axis representing the loss and the x-axis representing the weight value. The goal is to reach the lowest point in the graph (the global minimum). However, there are valleys in the graph where gradient descent can get stuck—these are local minima that are not the true global minimum. Now, imagine we add another weight, increasing the dimension of the graph by one. The graph is now three-dimensional. You can think of the loss surface as a two-dimensional valley, and the local minimum you were previously stuck in now has another dimension attached to it. This dimension is sloping downward (it's a saddle point), meaning you can escape the local minimum via this newly added dimension.

In general, the more dimensions you add, the higher the likelihood that a local minimum is not a true local minimum. There will likely be some dimensions that slope downward, allowing gradient descent to escape to lower minima.

Now, points 1 and 2 are not disconnected—they are two sides of the same coin. While the model is trying out different structures that don't affect its loss (point 1), gradient descent is roaming around the local minima without changing the loss (point 2). At some point, it may find a path out by discovering a dimension that slopes downward—a 'dimensional alleyway' out o...