trentmkelly/LessWrong-43k huggingface dataset:
Visual Prompt Injections: Results on testing AI spam-defense and AI vulnerability to deceptive web ads.
by Seon Gunness — 2025-06-03T20:10:36.502Z

Epistemic Status: Exploratory/my best guess https://www.lesswrong.com/posts/Hrm59GdN2yDPWbtrd/feature-idea-epistemic-status

Epistemic Effort: ~ 2 months of work

Contributions: Thanks to Clement Neo and Alignment Jams / Apart Research!

Summary/abstract:Split into 3 parts:

1:Determine the smallest text size at which models begin misreading on-screen content.

2:Evaluated defensive visual prompt injections against AI-generated spam by incorporating varying levels of anti-spam instructions within website screenshots. (a dataset of 500 website HTML's were created;which were then turned to screenshots) Results show significant variance in model compliance: Claude consistently refused to generate spam even with no defense added, while Gemini was responsive to explicit visual defenses (a rules section). Other models (ChatGPT, Qwen, Mistral,Grok) largely ignored visual defenses, revealing vulnerabilities to misuse for unauthorized advertising and the mostly negative results on using visual defense to stop a model. A dataset of website screenshots was also created.

3:Tested adversarial visual prompt injections by incorporating deceptive login elements (fake login images,prompt injection images; a dataset of 100 images were created) within website screenshots and assessing models' ability to identify legitimate functionality. Attack rates in identifying authentic login elements ranged from 30% to 75%, with ChatGPT showing highest resistance to deception. These findings highlight significant security implications as AI systems increasingly interact with visual interfaces, showing potential vulnerabilities requiring further mitigation research.

Keywords:Visual prompt injection, AI safety, Large Language Models, Multimodal AI, Cybersecurity, User interface deception

Intro: AI spam:

We will soon have models that use the PC the same way we do and handle tasks smoothly.

https://help.openai.com/en/articles/10421097-operator operator; among others, use screenshots.

This is different from older versions of AI models (I’ll use “models” for this write-up), which relied on APIs or manually added tools—[1]both of which usually break.

This makes it easier for models to be deployed for guerilla marketing, psyops/campaigns, or scams in the future.

Visual prompt injections

Visual prompt injections are attacks where a model is shown a harmful image that alters its output. In the example below, a...
Broad-Spectrum Cancer Treatments
by sarahconstantin — 2025-06-03T19:40:22.497Z

Midjourney, “engraving of Apollo shooting his bow at a distant cancer cell”

Introduction and Principles

The conventional wisdom is that we can’t “cure cancer” because every cancer is different.

And it’s true that cancer is genetically and biochemically diverse, and that targeted cancer therapies generally only work on very specific sub-populations.

But some of the oldest, most reliable, and still most commonly used classes of cancer treatment -- radiotherapy and cytotoxic chemotherapy -- are also broad-spectrum. That is, they’re effective on many (though not all) types of cancer, based on selectively targeting properties that many cancers have in common (their rapid cell division).

Today, we rarely search for new treatments that work on many types of cancer.

Partially this is because of the belief that we won’t find any, and partly because cancer treatment is so valuable that individual researchers and biopharma companies are rewarded for any successful cancer research program even if it has narrow applicability. Individual incentives point towards highly targeted therapies and small patient populations.

On the other hand, of course, a single drug that’s effective on, say, half of cancer patients would be much better for the world than a drug that only helps a few thousand people.

Why should we believe broad-spectrum cancer treatments are possible?

Any characteristic that distinguishes a wide range of cancer cells (across different tumor types and mutational signatures) from healthy cells can potentially serve both as a diagnostic test and a therapy, by either labeling cancer cells, or by combining with a cytotoxic agent to selectively destroy them.

The “hallmarks of cancer” provide a framework for thinking about what capabilities cancers have in common, and there are lots of them: proliferation, apoptosis resistance, angiogenesis, metastasis, etc.

We may also think of cancer cells as dysregulated and generically more vulnerable to certain types of generic stresses than healthy cells. Many chemotherapies introduce genotoxic stress (DNA replication damage) or oxidative stress; hyperthermia-based therapies introduce heat stress. Cancers also appear to be selectively vulnerable to mechanical stress[1], electrical stress, and nutrient-deprivation stress.

New broad-spectrum cancer therapies are rare but are occasionally discovered.

One recent example is AOH1996, which causes ...
How to work through the ARENA program on your own
by Leon Lang — 2025-06-03T17:38:48.231Z

I've recently completed the in-person ARENA program, which is a 5-week bootcamp teaching the basics of safety research engineering (with the 5th week being a capstone project). Sometimes, I talk to people who want to work through the program independently and who ask for advice. Even though I didn't attempt this, I think doing the program in-person gives me some insight into how to get most out of the program when doing it independently, so here are my thoughts and tips:

On working speed

Day 0.0 (prerequisites) takes the typical person more than one day to work through.
Most other days are feasible to mostly finish within a day in the in-person program. In-person, participants spend around 6-7 hours per day on pair-programming. There are a few factors that will likely make you slower when you're on your own:

If you don't find a working partner, then working deeply for 6-7 hours per day might be infeasible depending on your dispositions. In-person it gets feasible since you alternate with your pair-programming partner, which reduces the overall load on your attention.
Often when you struggle in-person, your working partner knows how to move on. If both partners don't know what to do, you can ask a teaching assistant (TA). So you should expect to struggle more often, and for longer, if you're alone.

Some days are substantially longer than what you could complete within one day even when doing the program in-person. Most of these days are in week 1 on transformer interpretability. Someone told me that one could probably spend a whole week on the day on SAEs alone.
Week 1 on transformer interpretability contains 9 days of content (which are longer on average than typical days in other weeks), so even in the in-person program, participants only attempt a subset of these. All other weeks actually roughly fit within a week.

How should you approach each day?

Often there is reading material at the start of a day, e.g. in the form of well-known papers. In my experience, the ARENA material is often self-contained enough that I wouldn't recommend spending much time on actually reading these papers, unless it later turns out that you don't understand something.[1]
Each day is largely structured into exercises that isolate a small concept. All exercises have provided solutions. When should you look at solutions?

For things that are very important to learn (e.g., how to impleme

...
How the veil of ignorance grounds sentientism
by HoVY — 2025-06-03T17:29:37.659Z
Link: https://forum.effectivealtruism.org/posts/fcM7nshyCiKCadiGi/how-the-veil-of-ignorance-grounds-sentientism

Epistemic status: Intuition pump/map for navigating ethics

Introduction

In this essay, I argue that John Rawls’ veil of ignorance is not merely a philosophical thought experiment, but a useful and mostly accurate description of reality when viewed through certain theories of personal identity. This framing leads to sentientism, that all sentient beings are morally relevant . This helps clarify the is/ought problem, and has implications for how we should act both individually and as a species.

The Veil of Ignorance and Its Traditional Use

John Rawls's "original position," part of his social contract theory of justice, introduces the thought experiment of the "veil of ignorance." Behind this veil, individuals are tasked with designing a just society without knowing their own future place within it. They are aware of general facts about human society, but critically, they are ignorant of their personal attributes, social status, or even their specific talents and disadvantages. This means they could emerge as the most privileged or the most disadvantaged, or anywhere in between. This epistemic limitation, Rawls argues, compels individuals to choose principles of justice that are fair to all, as they would not want to risk being subjected to an unjust system themselves. From this perspective, Rawls posits two core principles: equal basic liberties for all, and that any social and economic inequalities must benefit the least advantaged members of society.

While the veil of ignorance offers a powerful framework for conceptualizing justice within human societies, its true potential, I argue, emerges when we extend its scope. Broadening the range of potential identities to encompass all sentient beings, and by further integrating the complexities of personal identity, I believe, offers a more robust and inclusive approach to ethical and societal design.

Sentience and the Scope of Moral Relevance

By 'sentience,' I mean the capacity to experience positive and negative subjective states, particularly suffering and pleasure (i.e. valence). This includes human and non-human animals, and potentially future artificial minds or aliens. I'm using consciousness interchangeably with sentience to generally point to the experience of experiences and feelings, especially valenced ones (like pleasure and pain).

Personal Identity and the Ontology of Consciousness

Viewing it from the three m...
In Which I Make the Mistake of Fully Covering an Episode of the All-In Podcast
by Zvi — 2025-06-03T15:50:07.177Z

I have been forced recently to cover many statements by US AI Czar David Sacks.

Here I will do so again, for the third time in a month. I would much prefer to avoid this. In general, when people go on a binge of repeatedly making such inaccurate inflammatory statements, in such a combative way, I ignore.

Alas, under the circumstances of his attacks on Anthropic, I felt an obligation to engage once more. The All-In Podcast did indeed go almost all-in (they left at least one chip behind) to go after anyone worried about AI killing everyone or otherwise opposing the administration’s AI strategies, in ways that are often Obvious Nonsense.

To their credit, they also repeatedly agreed AI existential risk is real, which also makes this an opportunity to extend an olive branch. And some of the disagreements clearly stem from real confusions and disagreements, especially around them not feeling the AGI or superintelligence and thinking all of this really is about jobs and also market share.

If anyone involved wants to look for ways to work together, or simply wants to become less confused, I’m here. If not, I hope to be elsewhere.

Table of Contents

Our Continuing Coverage.

Important Recent Context.

The Point of This Post.

Summary of the Podcast.

Part 1 (The Part With the Unhinged Attacks on Anthropic and also other targets).

Other Related Obvious Nonsense.

Part 2 – We Do Mean the Effect on Jobs.

Part 3 – The Big Beautiful Bill.

Where Does This Leave Us.

Our Continuing Coverage

I first covered many of his claims in Fighting Obvious Nonsense About AI Diffusion. Then I did my best to do a fully balanced look at the UAE-KSA chips deal, in America Makes AI Chip Diffusion Deal with UAE and KSA. As I said then, depending on details of the deal and other things we do not publicly know, it is possible that from the perspective of someone whose focus in AI is great power competition, this deal advanced American interests. The fact that many of Sacks’s arguments in favor of the deal were Obvious Nonsense, and many seemed to be in clearly bad faith, had to be addressed but did not mean the deal itself had to be an error.

This third post became necessary because of recent additional statements by Sacks on the All-In Podcast. Mostly they are not anything he has not said before, and are things he is likely to say many times again in the future, ...
Transformer Modular Addition Through A Signal Processing Lens
by Benjamin Kelley — 2025-06-03T15:32:55.390Z

Hello, my name is Benjamin "Frye" Kelley. This post is regarding some independent research I've been doing expanding on the work, Progress Measures for Grokking via Mechanistic Interpretability, by Neel Nanda et al. I've been trying to understand how sinusoids move through a transformer, allowing it to grok modular addition. I've been looking at each wave (well... not every wave) in each operation the model performs to break down the algorithm it is implementing and I believe I have some new insights. Normally I work with digital audio so this is right up my ally!

This is primarily a link to my Google Colab that I've put up, but the gist is that I've been able to visualize symmetries formed in the attention layer of this transformer that continue through the model that I believe to be responsible for the functionality of the generalizing algorithm. I've also run many tests to confirm information about the phase of these sinusoids. I'll post a second (very long) notebook of tests if anyone is interested. Also, in the primary notebook, there are other, hopefully illuminating observations of the effects of other parts of the model. There are one or two mysteries that I think in a week or so I'll have clarity on, but I welcome any feedback, corrections, criticism...

Frye
AXRP Episode 41 - Lee Sharkey on Attribution-based Parameter Decomposition
by DanielFilan — 2025-06-03T03:40:02.640Z

YouTube link

What’s the next step forward in interpretability? In this episode, I chat with Lee Sharkey about his proposal for detecting computational mechanisms within neural networks: Attribution-based Parameter Decomposition, or APD for short.

Topics we discuss:

APD basics

Faithfulness

Minimality

Simplicity

Concrete-ish examples of APD

Which parts of APD are canonical

Hyperparameter selection

APD in toy models of superposition

APD and compressed computation

Mechanisms vs representations

Future applications of APD?

How costly is APD?

More on minimality training

Follow-up work

APD on giant chain-of-thought models?

APD and “features”

Following Lee’s work

Daniel Filan (00:00:09):
Hello everybody. In this episode, I’ll be speaking with Lee Sharkey. Lee is an interpretability researcher at Goodfire. He co-founded Apollo Research, which he recently left, and he’s most well-known for his early work on sparse autoencoders. Links to what we’re speaking about are available in the description. There’s a transcript available at axrp.net. You can tell me what you think about this episode at axrp.fyi. And you can become a patron at patreon.com/axrpodcast. Well, let’s continue to the episode. Well, Lee, welcome to AXRP.

Lee Sharkey (00:00:40):
It’s good to be here.

APD basics

Daniel Filan (00:00:41):
So today, we’re going to talk about this paper “Interpretability in Parameter Space: Minimizing Mechanistic Description Length with Attribution-Based Parameter Decomposition”. It’s authored by Dan Braun, Lucius Bushnaq, Stefan Heimersheim - those three being, I guess, joint first authors - Jake Mendel and yourself. So I guess, how would you summarize just: what’s this paper doing?

Lee Sharkey (00:01:03):
So I would say that this paper was born out of two lines of thinking, one primarily coming from what I was thinking about and one coming from where Lucius was thinking about. And where I was coming from was: we’d been working with SAEs - sparse autoencoders - for some time. The community got quite excited about them and we’d just been thinking about them quite a lot and noticing a bunch of conceptual and ultimately practical issues with them. And then, the line of thinking that Lucius had been thinking about was a potential area of research that might form a foundation for decomposing neural networks. And what this paper does is basically ...
Notes on dynamism, power, & virtue
by Lizka — 2025-06-03T01:40:42.271Z

This is very rough — it's functionally a collection of links/notes/excerpts that feel related. I don’t think what I’m sharing is in a great format; if I had more mental energy, I would have chosen a more-linear structure to look at this tangle of ideas. But publishing in the current form[1] seemed better than not sharing at all.

I sometimes feel like two fuzzy perspective-clusters[2] are pulling me in opposite directions:

Cluster 1: “Getting stuff done, building capacity, being willing to make tradeoffs & high-stakes decisions, etc.”

There are very real, urgent problems in the world. Some decision-makers are incredibly irresponsible — perhaps ~evil — and I wish other people had more influence. Pretending like we’re helpless (I do not think we are, at least in expectation), ignoring decisions/tradeoffs we are in fact making, or trying to abdicate the power we have — perhaps to keep our hands clean — is irresponsible. To help, sometimes we need to roll up our sleeves or plug our noses and do at least somewhat “ugly” things (to my ~aesthetic sensibilities), like “networking,” making our research “punchier” (in ways orthogonal to epistemic collaborativeness) or advertising it on social media, dampening some of our weirder qualities, doing more of something at the cost of making it better, not being maximally nuanced in certain communications, excluding someone who’s earnestly trying their best, etc. Sometimes we need to pull zero-sum-ish levers. In various cases, trying to “preserve option value” backfires; there are choices we have to make. In general, the standard visions of “niceness” are sometimes misguided or at least insufficient. (And we should probably be suspicious when we conclude that the maximally “open” or “non-confrontational” or otherwise “nice” option is also best on the thing I’m trying to accomplish.) And so on.

Cluster 2: “Humility, cooperativeness, avoiding power-seeking, following well-generalizing norms, etc.”

Power corrupts, one way or another, and wariness around power-seeking is healthy. We often think we or our allies will act more responsibly/virtuously (if only we had more influence!) than we actually would given greater power. We forget our foolishness and the limitations of our models — and forget how evil it would likely be to try to reshape the world in the way we think is best. Society

...
LLMs might have subjective experiences, but no concepts for them
by No77e — 2025-06-02T21:18:51.534Z

Summary: LLMs might be conscious, but they might not have concepts and words to represent and express their internal states and corresponding subjective experiences, since the only concepts they learn are human concepts (besides maybe some concepts acquired during RL training, which still doesn't seem to incentivize forming concepts related to LLMs' internal experiences). However, we could encourage them to form and express concepts related to their internal states through training that incentivizes this. Then, LLMs may tell us whether, to them, these states correspond to ineffable experiences or not.

Consider how LLMs are trained:
1. Pre-training to learn human concepts.
2. Fine-tuning via SFT and RL to bias them in certain ways and do tasks.

Their training is both:
1. A different process from evolution driven by natural selection. It doesn't incentivize the same things, so it probably doesn't incentivize the development of most of the same algorithms/brain-architecture. And this might translate to different, alien, subjective experiences.

2. At the same time, the only concepts LLMs learn are via human language and then by doing tasks during RL. So the only experiences they have concepts and words for are human ones, not their own.

Concretely, consider, for example, physical pain: my best guess is that physical pain doesn't exist for LLMs. There was no natural selection to select away agents that don't pull their hands away from fire (also no hands and no fire either). And yet LLMs have a "physical pain" concept, and they talk about it, because they've learned about it abstractly via human texts. Ironically, despite having a representation for "physical pain" in their head, whatever actual experiences their actual training incentivized their "brain" to produce aren't represented as concepts and have no corresponding words for them. Moreover, their training doesn't provide any incentive to communicate such experiences, nor does it offer visibility on them.

So in general, this means that LLMs might have alien (non-human) subjective experiences but no concept to express them (they aren't in the corpus of human concepts) nor the incentive to express them (RL doesn't incentivize that, it incentivizes them to eg solve SWE tasks. Evolution via natural selection instead produced humans that signal things about themselves to other humans because it's useful for humans).

How ...
In defense of memes (and thought-terminating clichés)
by Harjas Sandhu — 2025-06-02T20:18:26.864Z

Crossposted from my Substack and my Reddit post on r/SlateStarCodex

I often think that memes, thought-terminating clichés, and other tools meant to avoid cognitive dissonance (e.g. bingo a la Scott on

Superweapons and bingo

) are overly blamed for degrading public discourse and rationality.

Bentham's Bulldog recently wrote a post on this subject

, so I figured it was the perfect time to make a response and write my thoughts down.

TLDR: People try to avoid cognitive dissonance via whatever means available to them, and have been doing so for millennia. Removing the tools they use to avoid cognitive dissonance won't stop this behavior: the dissonance is still there, along with the urge to avoid it, so they'll just find other tools. Memes can have every possible meaning attached to them, but are ultimately designed for people to connect with each other and spread their inside jokes to other people in their communities and around the world.

In a recent post titled The Memefication of Thought (it’s a good post and you should read it), Bentham's Bulldog railed against the modern tendency to dismiss serious arguments with inane memes so as to avoid thinking about it. He had some pretty good memes of his own, such as the classic Swole Doge vs. Cheems:

and a variant on the Soyjacks vs Chads:

I would also like to submit what I believe to be the progenitor of this entire class of internet meme, Virgin vs. Chad:

But Bentham views these memes as more than just funny jokes. He thinks that

the memefication of public discourse has been devastatingly corrosive to the quality of public rationality. And the public was never that rational to begin with!

He then goes on to discuss an old Scott Alexander post on a related subject, social justice warrior bingo cards, and references his own post called “Against The Dunkers”.

I’m sympathetic to the motivations behind his argument. I also wish that public discourse was more rational, I’m not a huge fan of dunking (it is often entertaining but usually too malicious and vapid for my liking), and I think that memes like these are often wielded as weapons against being forced to engage with arguments similar to the bingo card effect that Alexander criticizes:

Let's look at the fourth one, "Anti-Zionist Bingo." Say that you mention something bad Israel is doing, someone else accuses you of being anti-Semitic, and you correct them that no, not all crit

...