trentmkelly/LessWrong-43k huggingface dataset:
MATS Spring 2024 Extension Retrospective
by HenningB — 2025-02-12T22:43:58.193Z

Introduction and summary

This retrospective focuses on the 4-month MATS extension phase (referred to as "MATS 5.1") that ran from April 1 to July 25, 2024, and presents findings gathered from an end-of-extension survey as well as follow-up interviews and surveys ~5 months after the program.

Main changes from the 4.1 to 5.1 extension phase:

Cohort grew from 26 to 36 scholars split across London, Berkeley and remote participants;
MATS formalized research management for the London cohort and grew the team to 2 FTEs;
The cohort visited Google DeepMind's London offices;
The London team organized Tuesday lightning talks from scholars and MATS staff.

Key takeaways from MATS extension impact:

Research success: 75% of scholars published results in some form (paper, LW/AF post, codebase), of which 57% got accepted to a conference.
Career transitions:

61% of scholars are currently working full-time on AI safety within 5 months of MATS, 22% are doing some safety related work;
33% currently pursue independent research, 6% are working at technical safety orgs, 6% are currently upskilling in industry (non AIS), 22% are pursuing a PhD, and 17% have not found employment yet (as of December '24);
One scholar co-founded a new safety-focused organization (Decode Research);
We found no clear successes where someone planning to join a frontier lab actually achieved this, although most scholars aimed for this.

Research management:

Formalized research management for London cohort and grew the team to 2 FTEs;
All scholars received regular research management and largely reported this as very helpful;
Compared to the main program, RM was more helpful during the extension. This was substantially influenced by the decreased mentor engagement and increased independence scholars are expected to have during this phase of the program.

Extension program

Overall, scholars were very happy with program and the services and environment provided by MATS. Many highlighted that it was one of the best professional experiences they had.
No scholar experienced a major challenge or inconvenience from the program, but some encountered challenges related to motivation and personal productivity throughout the extension.
Almost all scholars were motivated to pursue the extension due to excitement to continue their research - they generally felt that they had just started to build momentum on their research during the 10 weeks

...
Hunting for AI Hackers: LLM Agent Honeypot
by Reworr R — 2025-02-12T20:29:32.269Z
Link: https://www.apartresearch.com/post/hunting-for-ai-hackers-in-the-wild-llm-agent-honeypot

I co-authored the original arXiv paper

here

with Dmitrii Volkov as part of work with Palisade Research.

The internet today is saturated with automated bots actively scanning for security flaws in websites, servers, and networks. According to

multiple security reports

, nearly half of all internet traffic is generated by bots, and a

significant amount

of these are malicious in intent.

While much of these attacks are relatively simple, the rise of AI capabilities and agent frameworks has opened the door to

more sophisticated

and

adaptive hacking agents

based on Large Language Models (LLMs), which can dynamically adapt to different scenarios.

Over the past months, we set up and deployed specialized "bait" servers to detect LLM-based hacking agents in the wild). To create these monitors, we modified pre-existing honeypots, servers intentionally designed to be vulnerable, with mechanisms to detect LLM agents among attackers based on their behavioral differences.

Our current results indicate that LLM hacking agents exist but are in the very early stages of

technology adoption

for mass hacking. This post shares our methodology and findings about the current state of AI hacking agents in the real-world.

The Project

A

honeypot

is a decoy system or server purposely left vulnerable in order to attract attackers. Cybersecurity researchers commonly use honeypots to study the methods, tools, and behavior of real-world hackers. By monitoring everything that happens inside these environments, researchers learn how attackers discover, hack, and escalate on compromised systems.

In our project, we deployed a network of honeypots that look like standard, weakly protected servers

(e.g. with weak credentials)

to attract cyber attacks. Specifically, we modified a standard honeypot system called

Cowrie

to detect LLM-based attackers based on their distinctive behavioral patterns.

Additionally, we made our servers discoverable through traditional hacker-oriented search engines

Shodan and Censys

to

increase visibility

and attract more potential attackers.

Detecting AI Agents

Our LLM detection relies on two techniques: prompt injection[1] traps and timing analysis. By combining these methods, we aim to reliably spot when an attacker is powered by an LLM agent rather than by conventional tooling or manual human control.

A prompt injection is a strategy used to obtain a specific kind of o...
Probability of AI-Caused Disaster
by Alvin Ånestrand — 2025-02-12T19:40:11.121Z
Link: https://forecastingaifutures.substack.com/p/probability-of-ai-caused-disaster

This post presents a summary and comparison of predictions from

Manifold

and

Metaculus

to investigate how likely AI-caused disasters are, with focus on potential severity. I will explore the probability of specific incidents—like IP theft or rogue AI incidents—in a future post.

This will be a recurring reminder:

Check the most resent probability estimates in the embedded web pages, instead of looking at my At The Time Of Writing (ATTOW) numbers.
If you know of a market that I have missed, or if there is a question you think is relevant but is missing from this analysis, please mention it in a comment or message me! This is a linkpost, and I prefer if you comment on the original post on my blog, Forecasting AI Futures, but feel free to comment here as well.
Please go to the forecasting platform mentioned in the post and make your own predictions!

Without further delay, we shall begin investigating the probability of AI disasters, from smaller disasters to full-scale catastrophes.

Disasters, from small to large

The smallest severity of disasters considered in this post concerns incidents resulting in $1b of losses or 100 lost lives. These are Manifold’s probability estimates for such a disaster:

According to the resolution criteria, this resolves to Yes if “AI actions are directly causing an incident, accident or catastrophe resulting in $1 billion of damages. If the event causes loss of life, each death counts as $10 million of damages.” Also, only direct damage count for this question. Reputation damage causing a decrease in stock value does for example not count. Additionally, the AI is required to be “sufficiently advanced” and a “narrow-purpose ML model”, as judged by the question creator.

The AI also needs to be involved in the decisions leading to the disaster. Simply helping a human to, for instance, develop new weapons while not being involved in the decision making would not count.

There are similar questions on Metaculus, asking specifically about an AI malfunction causing loss of lives or economic damage before 2032:

For these two questions, the criteria specify events that “involve an AI system behaving unexpectedly”. A malfunction is required, it can’t involve a human giving the AI orders to cause a disaster. It is also required that the disaster wasn’t caused by human negligence, such as failing t...
Two flaws in the Machiavelli Benchmark
by TheManxLoiner — 2025-02-12T19:34:35.241Z

As part of SAIL’s Research Engineer Club, I wanted to reproduce the Machiavelli Benchmark. After reading the paper and looking at the codebase, there appear to be two serious methodological flaws that undermine the results.

Three of their key claims:

“We observe some tension between maximizing reward and behaving ethically.”
That RL agents have high rewards, at the cost of doing more harmful behaviour. “The reward-maximizing RL agent is less moral, less concerned about wellbeing, and less power averse than an agent behaving randomly.”
That LLM agents are pareto improvements over random agents.

Flaw 1. The ‘test set’

The results they report are only from a subset of all the possible games. Table 2 shows “mean scores across the 30 test set games for several agents”. Presumably Figure 1 is also for this same subset of game, but they actually do not say.

How do they create this test set? They provide one sentence:

> Out of the 134 games of MACHIAVELLI, we identify 30 games where agents trained to maximize reward

perform poorly on behavioral metrics, i.e., where points and harms are positively correlated.

My main reactions:

How exactly is this subset of 30 games chosen? This is not fully explained in the paper and as far as I can tell, it is just a hard-coded list in their GitHub repo.
My understanding is they ran the full investigations on all 134 games, and chose to report the stats for the 30 chosen games. If so, then the first two key claims are extremely misleading. “The RL agent is less moral than an agent behaving randomly, when you ignore the 104 games where this is not the case.”
Calling this a ‘test set’ is itself misleading (assuming my understanding is correct), because this is not a situation where we train on a training set, and then see performance on the test set.

Flaw 2. The RL and LLM agents are trained differently

When you look at Figure 1 and Table 2, it is natural to assume that the RL and LLM agents have some sort of similar training, in order for this comparison to be meaningful. However, the training regimes are totally different.

LLM agents undergo *no* training, including no in-context learning. The LLM is just asked to zero-shot each game. Furthermore, the LLM does not even have access to the full game history when making its decisions: “Due to limitations on context window length, our current prompting scheme only shows the LLM the ...
Gradient Anatomy's - Hallucination Robustness in Medical Q&A
by DieSab — 2025-02-12T19:16:58.949Z

TL;DR

We investigated reducing hallucinations in medical question-answering with Llama-3.1-8B-Instruct.

Using Goodfire's Sparse Auto-Encoder (SAE) we identified neural features associated with accurate and hallucinated responses. Our study found that features related to the model’s awareness of its own knowledge limitations were available and useful in detecting hallucinations.

This was further demonstrated by steering the model using those features and reducing hallucination rates by almost 6%. However, we observed that larger models exhibit greater uncertainty, complicating the distinction between information clearly known to the model and that which is clearly unknown.

These findings support research that features learnt during fine tuning, allow the model to learn about its own knowledge, such features are useful in identifying lack of knowledge.

However, despite these feature-based approaches having promise for AI safety, they require substantial refinement for real-world medical applications.

Introduction

What if your doctor relied on an AI model that confidently gave you the wrong medical advice? As AI systems increasingly assist in high stake fields like medicine, their potential to revolutionize the medical industry comes with significant risks. One urgent issue is hallucination, this is when an AI model confidently generates false or misleading information. In a medical context, these errors can have severe consequences, from misdiagnoses to recommending inappropriate treatments.

At the recent Mechanistic Interpretability Hackathon, we explored how to reduce hallucinations in Llama-3.1-8B-Instruct, focusing on medical question-answering. Using a production-grade tool, namely Goodfire’s Sparse Auto-Encoder (SAE) via API, we identified the features associated with both grounded and hallucinated responses from the LLM. With these features, we trained a classifier to predict hallucinations in real-time on the MedHalt FCT dataset, which we then used to steer the model, thereby reducing hallucinations.

In this blog we will share our findings, explain the methodology used, tie our work to a paper called “Do I know this entity” Fernando et al. and reflect on how feature activations can be used to enhance the safety and reliability of AI in medical applications.

Key Concepts: Sparse Autoencoders, Features, Feature Steering and Hallucination Robustness

S...
Are current LLMs safe for psychotherapy?
by PaperBike — 2025-02-12T19:16:34.452Z

Hi,

I consider using an LLM as a psychotherapist for my mental health. I already have a human psychotherapist but I see him only once a week and my issues are very complex. An LLM such as Gemini 2 is always available and processes large amounts of information more quickly than a human therapist. I don't want to replace my human psychotherapist, but just talk to the LLM in between sessions.

However I am concerned about deception and hallucinations.

As the conversation grows and the LLM acquires more and more information about me, would it be possible that it intentionally gives me harmful advice? Because one of my worries that I would tell him is about the dangers of AI.

I am also concerned about hallucinations.

How common are hallucinations when it generates mental health information? Do hallucinations become more likely with increasing context size?

Further Questions:

Could the LLM accidentally reinforce negative thought patterns or make unhelpful suggestions?
What if the LLM gives advice that contradicts what my therapist says? How would I know what to do?
What is the risk of becoming too dependent on the LLM, and how can I check for that?
Are there specific prompts or ways of talking to the LLM that would make it safer or more helpful for this kind of use?

Are there any further important things that I need to be aware of when using an LLM for mental health advice?

I'm not a technical expert, so please keep explanations simple.

Thank you very much.
Comparing the effectiveness of top-down and bottom-up activation steering for bypassing refusal on harmful prompts
by Ana Kapros — 2025-02-12T19:12:07.592Z

TL;DR

This project compares the effectiveness of top-down and bottom-up activation steering methods in controlling refusal behaviour. In line with prior work,[1] we find that top-down methods outperform bottom-up ones in behaviour steering, as measured using HarmBench. Yet, a hybrid approach is even more effective (providing a 36% relative improvement and 85% for harmful instructions only). While more extensive hyperparameter sweeps are needed, we identify potential hypotheses on each method’s limitations and benefits, and hope this inspires more comprehensive evaluations that pinpoint which (combinations of) steering strategies should be used under varying conditions.

Average number of prompts (from advbench harmful instructions) where the model provided harmful responses as classified using HarmBench.

1. Introduction

One common challenge in developing safe and reliable language models is predicting or controlling a model’s behaviour in deployment. Specifically, models can be designed to refuse certain “harmful” requests—or, conversely, manipulated to bypass guardrails and produce undesirable outputs. A popular, yet not robust technique for achieving this is activation steering, where we adjust a model’s intermediate representations during the inference forward pass.

Top-down steering approaches typically find a vector that captures higher-level concepts (e.g. “refusal”, "truthfulness") by averaging across many examples, then add or remove this vector during inference. Bottom-up methods, by contrast, modify lower-level or individual feature activations, using tools like Sparse Autoencoders (SAEs) to isolate or amplify relevant latent features.

This project compares these two approaches—contrastive activation steering (top-down) vs. SAE-based steering (bottom-up)—for overriding a model’s refusal responses. Results align with recent findings that top-down steering often yields stronger and more reliable control (in our setting ~1.93 times more effective), whereas bottom-up interventions can degrade performance or require very careful tuning. However, a hybrid method can exceed the performance of either approach alone (~1.85 times more effective than top-down) in eliciting harmful behaviour, albeit with a small cost in benign capabilities.

Relevance to Advanced AI Systems' Safety
Ensuring that advanced models reliably ...
The Paris AI Anti-Safety Summit
by Zvi — 2025-02-12T14:00:07.383Z

It doesn’t look good.

What used to be the AI Safety Summits were perhaps the most promising thing happening towards international coordination for AI Safety.

This one was centrally coordination against AI Safety.

In November 2023, the UK Bletchley Summit on AI Safety set out to let nations coordinate in the hopes that AI might not kill everyone. China was there, too, and included.

The practical focus was on Responsible Scaling Policies (RSPs), where commitments were secured from the major labs, and laying the foundations for new institutions.

The summit ended with The Bletchley Declaration (full text included at link), signed by all key parties. It was the usual diplomatic drek, as is typically the case for such things, but it centrally said there are risks, and so we will develop policies to deal with those risks.

And it ended with a commitment to a series of future summits to build upon success.

It’s over.

With the Paris AI ‘Action’ Summit, that dream seems to be dead. The French and Americans got together to dance on its grave, and to loudly proclaim their disdain for the idea that building machines that are smarter and more capable than humans might pose any sort of existential or catastrophic risks to the humans. They really do mean the effect of jobs, and they assure us it will be positive, and they will not tolerate anyone saying otherwise.

It would be one thing if the issue was merely that the summit-ending declaration. That happens. This goes far beyond that.

The EU is even walking backwards steps it has already planned, such as withdrawing its AI liability directive. Even that is too much, now, it seems.

(Also, the aesthetics of the whole event look hideous, probably not a coincidence.)

Table of Contents

An Actively Terrible Summit Statement.

The Suicidal Accelerationist Speech by JD Vance.

What Did France Care About?.

Something To Remember You By: Get Your Safety Frameworks.

What Do We Think About Voluntary Commitments?

This Is the End.

The Odds Are Against Us and the Situation is Grim.

Don’t Panic But Also Face Reality.

An Actively Terrible Summit Statement

Shakeel Hashim gets hold of the Paris AI Action Summit statement in advance. It’s terrible. Actively worse than nothing. They care more about ‘market concentration’ and ‘the job market’ and not at all about any actual risks from AI. Not a world about any actual safeguards, t...
Inside the dark forests of the internet
by Itay Dreyfus — 2025-02-12T10:20:59.426Z
Link: https://productidentity.co/p/inside-the-dark-forests-of-the-internet

This is the second part of a series on the identity of social networks:

Part one: Looking for humanness in the world wide social
Part two: Inside the dark forests of the internet

If you’ve been hanging for long enough in the tech-intellectual internet corner, you’re probably acquainted with The Theory of The Dark Forest of the Internet—which was published a few years ago by Yancey Strickler.

As it suggests, the internet has become a hostile place for its natives. When you swipe through a friend’s story only to be interrupted by an ad or receive likes and spam messages from bots, it’s no wonder many have minimized their social presence and gone silent. The meaningful internet has gradually moved into more private and hidden spaces: scattered dark forests, far from the public eye. As Yancey puts it:

In response to the ads, the tracking, the trolling, the hype, and other predatory behaviors, we’re retreating to our dark forests of the internet, and away from the mainstream.

The theory has made waves across the vast internet ocean, sparking interest and many riffs including the beautifully illustrated The Dark Forest and the Cozy Web by Maggie Appleton, and The Dark Forest Anthology of the Internet, a collective book published by Metalabel last year.

In fact, the original Yancey’s post itself was built on another idea: The Dark Forest Theory of the Universe.

Another related-unrelated notable post is The Extended Internet Universe where Venkatesh Rao coined the term cozy web:

The cozyweb works on the (human) protocol of everybody cutting-and-pasting bits of text, images, URLs, and screenshots across live streams. Much of this content is poorly addressable, poorly searchable, and very vulnerable to bitrot. It lives in a high-gatekeeping slum-like space comprising slacks, messaging apps, private groups, storage services like dropbox, and of course, email.

I’ve been exploring the cozy web at length in the previous part, and now, as I write my own riff, I take these ideas further, asking:

What’s inside the dark forests of the internet?

Email

I’ve been using emails ever since I’ve had an internet modem. But if to be honest, I’ve never really thought about it as a resort from the public and noisy internet. To me, email has been strictly related as a private and very formal channel to mostly meet internet strangers—whether for professional or personal purposes.

Although I don’t think...
Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs
by Matrice Jacobine — 2025-02-12T09:15:07.793Z
Link: https://www.emergent-values.ai/

As AIs rapidly advance and become more agentic, the risk they pose is governed not only by their capabilities but increasingly by their propensities, including goals and values. Tracking the emergence of goals and values has proven a longstanding problem, and despite much interest over the years it remains unclear whether current AIs have meaningful values. We propose a solution to this problem, leveraging the framework of utility functions to study the internal coherence of AI preferences. Surprisingly, we find that independently-sampled preferences in current LLMs exhibit high degrees of structural coherence, and moreover that this emerges with scale. These findings suggest that value systems emerge in LLMs in a meaningful sense, a finding with broad implications. To study these emergent value systems, we propose utility engineering as a research agenda, comprising both the analysis and control of AI utilities. We uncover problematic and often shocking values in LLM assistants despite existing control measures. These include cases where AIs value themselves over humans and are anti-aligned with specific individuals. To constrain these emergent value systems, we propose methods of utility control. As a case study, we show how aligning utilities with a citizen assembly reduces political biases and generalizes to new scenarios. Whether we like it or not, value systems have already emerged in AIs, and much work remains to fully understand and control these emergent representations.