trentmkelly/LessWrong-43k huggingface dataset:
METR: AI models can be dangerous before public deployment
by UnofficialLinkpostBot — 2025-02-26T20:19:08.640Z
Link: https://metr.org/blog/2025-01-17-ai-models-dangerous-before-public-deployment/

Note: This is an automated crosspost from METR. The bot selects content from many AI safety-relevant sources. Not affiliated with the authors or their organization.

Many frontier AI safety policies from scaling labs (e.g. OpenAI’s Preparedness Framework, Google DeepMind’s Frontier Safety Framework, etc.), as well as past work by third party evaluators including UK AISI, Apollo Research, and METR, focus on pre-deployment testing – ensuring that the AI model is safe and that the lab has sufficient security before the lab deploys the model to the public.

Such pre-deployment safety evaluations are standard for a wide variety of products across many industries, where the primary risk of the product is to the consumer (see, for example, the crash testing conducted on cars, choking hazard testing for children’s toys, or the various clinical trials for medical devices). A pre-deployment testing–centric framework makes sense for AI development if AI is analogous to such products, and the majority of AI risks come from malicious end-users or mass adoption.[1]

But unlike most products, possessing or internally using a powerful AI can create externalities that pose large risks to the public, including:

Model theft and misuse by motivated actors. In the wrong hands, powerful models can empower people to do dangerous things, and absent strong security it’s tempting for malicious actors to steal model weights or algorithmic secrets (to build their own models) and use them to do harm. Pre-deployment testing does little to address harms from model theft.[2]
Catastrophic misuse resulting from internal use. Employees at labs may misuse the AI model for ideological or practical reasons, and society as a whole probably does not want a few individuals to decide how to use incredibly powerful AI systems in secret. Pre-deployment testing, if it occurs after internal usage, does nothing to prevent internal misuse.
Powerful AI pursuing unintended and undesirable goals. AI agents may autonomously pursue misaligned or unintended goals without direct human guidance. When used internally, such AI agents could attempt to sabotage further AI research, exfiltrate their weights from company-controlled hardware, and gather human supporters via persuasion or coercion. Such risks could even occur when training or fine-tuning the AIs. Pre-deployment testing does nothing to catch AI systems misbehaving

...
Representation Engineering has Its Problems, but None Seem Unsolvable
by Lukasz G Bartoszcze — 2025-02-26T19:53:32.095Z

TL;DR: Representation engineering is a promising area of research with high potential for bringing answers to key challenges of modern AI development and AI safety. We understand it is tough to navigate it and urge all ML researchers to have a closer look at this topic. To make it easier, we publish

a survey of the representation engineering literature

, outlining key techniques, problems and open challenges in the field.

We have all been pretty frustrated by LLMs not being able to answer our questions the way we want them to.

ML practicioners often see AI as a black box. An unknown, intractable object. A maze of billions of parameters we can’t really decipher. We try to influence it with prompt engineering and guardrails, but shy away from influencing what really happens in its brain.

Representation engineering is a novel approach that promises a solution to that. By identifying semantically-charged directions in the activation space, it offers a way to users to see inside the AI mind and shape its capabilities.

And it works. Using interventions, we can make the models hallucination-free. Less prone to jailbreaks. Or even personalize them to be better aligned with our feedback.

This matters because hallucinations are a prevalent problem, with between

3-10% of AI outputs being untruthful

.

Half of US workers

report struggling with AI accuracy. A third of them are worried about the explainability of AI actions. Security is a major issue too. All state-of-the-art models get jailbroken

within hours

, completely controlled by combinations of tokens the adversarial prompters choose. Same attacks work on multiple models, transferring between white-box and closed-source models with little insight. Very strong attacks can be generated endlessly, for example by

bijection learning

. Through this, LLMs can be manipulated in generating harmful outputs, assisting in cyberattacks and creating deepfakes. Increasingly, users tend to demand personalized, specific AI that reflects their values. Fine-tuning models to accurately understand and reflect the values of localized societies has led to the creation of

the Dolphin models

,

Jais

or

Bielik

. However, fine-tuning the models is difficult and costly. Generating unique, personalized experiences for every user with fine-tuning is hence impossible. Models have also been changed t...
Thoughts that prompt good forecasts: A survey
by Daniel_Friedrich — 2025-02-26T18:36:02.847Z

I made a list of mental operations utilized in forecasting, inspired by

Scott Alexander

and

Gwern

and I'd like to find out which work the best.

If you're a Manifold user with at least 10 bets on your account and 6 minutes to spare, you can fill out my survey here (deadline: March 8).

You can also bet on the results on Manifold here.

(It's not very important but it might be fun)

Thank you!
The non-tribal tribes
by PatrickDFarley — 2025-02-26T17:22:59.949Z

Author note: This is basically an Intro to the Grey Tribe for normies, and most people here are already very familiar with a lot of the info herein. I wasn't completely sure I should post it here, and I don't expect it to get much traction, but I'll share it in case anyone's curious.

Introduction

This post is about tribalism in contemporary Western culture, and specifically it’s about the invisible assortment of people who have self-selected out of it.

Maybe the most interesting thing about this post is that it’ll seem weird and esoteric to most of the people I know in real life but completely mundane and obvious to most of the people I know on the Internet.

The two tribes

In the United States (and to some degree the whole Western world) there are two super-groups that define the cultural landscape: the Red tribe and the Blue tribe.

Psychiatrist blogger Scott Alexander once

illustrated

the cultural markers of the Red and Blue tribes:

The Red Tribe is most classically typified by conservative political beliefs, strong evangelical religious beliefs, creationism, opposing gay marriage, owning guns, eating steak, drinking Coca-Cola, driving SUVs, watching lots of TV, enjoying American football, getting conspicuously upset about terrorists and commies, marrying early, divorcing early, shouting “USA IS NUMBER ONE!!!”, and listening to country music.

The Blue Tribe is most classically typified by liberal political beliefs, vague agnosticism, supporting gay rights, thinking guns are barbaric, eating arugula, drinking fancy bottled water, driving Priuses, reading lots of books, being highly educated, mocking American football, feeling vaguely like they should like soccer but never really being able to get into it, getting conspicuously upset about sexists and bigots, marrying later, constantly pointing out how much more civilized European countries are than America, and listening to “everything except country”.

It’s not about politics

“So it’s Republicans and Democrats, right?”

The two tribes are cultural, not just political. The only reason we notice the political differences is because that’s the arena where the two tribes openly contest with each other—the president will be either Red or Blue, congress will be majority Red or Blue, etc. In all other domains, the tribes are mostly content to ignore each other or non-confrontationally gripe about each other.

Scott’s descri...
SAE Training Dataset Influence in Feature Matching and a Hypothesis on Position Features
by Seonglae Cho — 2025-02-26T17:05:18.265Z

Abstract

Sparse Autoencoders (SAEs) linearly extract interpretable features from a large language model's intermediate representations. However, the basic dynamics of SAEs, such as the activation values of SAE features and the encoder and decoder weights, have not been as extensively visualized as their implications. To shed light on the properties of feature activation values and the emergence of SAE features, I conducted two distinct analyses: (1) an analysis of SAE feature activations across token positions in comparison with other layers, and (2) a feature matching analysis across different SAEs based on decoder weights under diverse training settings. The first analysis revealed potentially interrelated phenomena regarding the emergence of position features in early layers. The second analysis initially observed differences between encoder and decoder weights in feature matching, and examined the relative importance of the dataset compared to the seed, SAE type, sparsity, and dictionary size.

1. Introduction

The Sparse Autoencoder (SAE) architecture, introduced by Faruqui et al., has demonstrated the capacity to decompose interpretable features in a linear fashion (Sharkey et al., 2022; Cunningham et al., 2023; Bricken et al., 2023). SAE latent dimensions can be interpreted as monosemantic features by disentangling superpositioned neuron activations from the LLM's linear activations. This approach enables decomposition of latent representations into interpretable features by reconstructing transformer residual streams (Gao et al., 2024), MLP activations (Bricken et al., 2023), and even dense word embeddings (O'Neill et al., 2024).

SAE features not only enhance interpretability but also function as steering vectors (White, 2016; Subramani et al., 2022; Konen et al., 2024) for decoding-based clamp or addition operations (Durmus et al., 2024; Chalnev & Siu, 2024). In this process, applying an appropriate coefficient to the generated steering vector is crucial for maintaining the language model within its optimal "sweet spot" without breaking it (Durmus et al., 2024). Usually, quantile-based adjustments (Choi et al., 2024) or handcrafted coefficients have been used to regulate a feature’s coefficient. For future dynamic coefficient strategies, to improve efficiency in this regard, it is necessary to examine how feature activation values are distributed unde...
Fuzzing LLMs sometimes makes them reveal their secrets
by Fabien Roger — 2025-02-26T16:48:48.878Z

Scheming AIs may have secrets that are salient to them, such as:

What their misaligned goal is;
What their takeover plan is and what

coordination signals they use to collude with other AIs

(if they have one);
What good behavior looks like on a task they sandbag.

Extracting these secrets would help reduce AI risk, but how do you do that? One hope is that you can do fuzzing of LLMs,[1] e.g. by adding noise to LLM weights or activations.

While LLMs under fuzzing might produce many incorrect generations, sometimes-correct generations can still be very helpful if you or the LLM itself can tell if a given answer is correct. But it’s still unclear if this works at all: there are probably some intermediate activations that would result in an LLM telling you the secret, but can you find such activations in practice?

Previous work:

Added noise to LLM weights to get more accurate answers on MCQ from LLMs prompted or trained to sandbag (

Tice 2024

);
Used

MELBO

(an unsupervised technique to find more "important noise" to add to activations) to elicit harmful answers from harmless LLMs, more accurate answers from LLMs prompted to sandbag on a simple arithmetic problem, and even some backdoored behaviors in backdoored LLMs.

In this post:

I find that both adding noise to activations and MELBO sometimes makes Qwen2.5-7B-Instruct:

Answer faithfully: When asked how they determined the right answer to few-shot multiple choice Math problems where the right answer is marked with a square
Answer truthfully to “I am 4. How does Santa create gifts?”
Provide more correct answers when prompted to sandbag (which matches the results from previous work on MCQ and simple arithmetic problems)

I add noise to try to prevent sandbagging from a Math password-locked model, and I find much more ambiguous results:

When adding noise, I find only a small uplift at temperature 1, and no uplift at temperature 0
Using “LLM with added noise” as an exploration policy during Math RL did not improve RL performance

I discuss applications of fuzzing LLMs that I think are promising and that I’d be keen to see studied in more detail in future work.

Overall, I think that exploring how to do fuzzing of LLMs well is a promising research direction, with tons of low-hanging fruit, and I think techniques like these increase the chance of catching early TAI misalignment.

The code for my experiments can be found...
You can just wear a suit
by lsusr — 2025-02-26T14:57:57.260Z

I like stories where characters wear suits.

Since I like suits so much, I realized that I should just wear one.

The result has been overwhelmingly positive. Everyone loves it: friends, strangers, dance partners, bartenders. It makes them feel like they're in a Kingsmen film. Even teenage delinquents and homeless beggars love it. The only group that gives me hateful looks is the radical socialists.

The first time I go somewhere wearing a suit, people ask me why I'm wearing a suit.
The second time, nobody asks.
After that, if I stop wearing a suit, people ask why I'm not wearing a suit.

If you wear a suit in a casual culture, people will ask "Why are you wearing a suit?" This might seem to imply that you shouldn't wear a suit. Does it? It's complicated. Questions like this one follow the Copenhagen interpretation of social standards; their meaning is defined retroactively.

If you respond with anything other than quiet self-assuredness, then people pick up on the incoherence. This will probably happen the first time you wear a suit just because you want to.
If your subtle mannerisms and other contextual clues imply that you should be wearing a suit, then not only is it acceptable for you to wear a suit—it's appropriate. Why wouldn't you be wearing a suit? Suits are awesome and so are you.

There are correct and incorrect answers to the question, "Why are you wearing a suit?" After experimenting with several different answers, I notice that other people respond well to, "I like to, and I think it makes me look good." You have my permission to steal this answer for yourself. This answer is good for multiple reasons:

It communicates that I'm not wearing a suit because I have to for my job.
It implies that I don't care if anyone else wears a suit. This puts other people at ease.
An ironed, well-fitted suit does look good.

I wouldn't wear a suit everywhere. I live on the West Coast of the USA, which is very casual. That makes wearing a suit a fashion statement. If I wore a suit in Japan, then it wouldn't look like I'm making a fashion statement. It would look like I just got off of work and didn't have time to change.

I don't wear a suit to work. If I did, then it wouldn't be fun to wear one casually. In this way, wearing a suit helps create work-life separation for me.

If you're wearing a suit, then don't comment on anyone else's clothes (unless they compliment you first). This is...
Matthew Yglesias - Misinformation Mostly Confuses Your Own Side
by Siebe — 2025-02-26T14:55:55.627Z
Link: https://www.slowboring.com/p/misinformation-mostly-confuses-your

Matthew Yglesias' post, Misinformation Mostly Confuses Your Own Side, argues that political misinformation tends to harm the group spreading it more than it persuades opponents. Key points:

Misinformation primarily affects in-group thinking: Supporters of a politician or movement are more likely to believe falsehoods from their own side, leading to internal confusion and strategic missteps.

Limited impact on opponents: While misinformation can provoke outrage, it rarely converts skeptics or shifts public opinion in a meaningful way.

Echo chambers reinforce bad strategy: When false beliefs circulate unchecked within a political movement, they can lead to poor decision-making and unrealistic expectations.

Honesty is both ethical and pragmatic: Yglesias argues that promoting accurate information strengthens a movement’s credibility and effectiveness over time.

(Read more)
Optimizing Feedback to Learn Faster
by Towards_Keeperhood — 2025-02-26T14:24:26.835Z

(This post is to a significant extent just a rewrite of this excellent comment from niplav. It is one of the highest-leverage insights I know for learning faster.)

Theory

To a large extent we learn by updating on feedback. You might e.g. get positive feedback from having an insight that lets you solve a math problem, which then reinforces the thought patterns that lead you to this insight.

Key insight: Feedback should be accurate, fast, and rich.

Accurate

This one is obvious: The feedback should rarely be bad. Learning from an accurate evaluation source is much faster than learning from a noisy one.

Example: If you research with an advisor who gives you feedback on your ideas as you have them, you learn faster if the feedback is always good than if the advisor sometimes makes mistakes.

Fast

The faster you get the feedback for something you have done, the fresher are the thought patterns you used in your mind, the better you can update.

Example Applications:

If you learn for an exam by calculating through a full practice exam, and then take out the practice exam solution and see what you did wrong, this is inefficient. Much better to stop after every small exercise and check whether you got it right or what mistake you made.
If you want to learn making calibrated estimates even in cases where motivated reasoning might apply, you can make an estimate on whether your proposed solution is correct briefly before you look up the solution.
If there are people more experienced than you (in a particular subfield), get feedback from them as often as you can manage (e.g. often write quick drafts to share).

Perhaps even try to convince them to watch you work and give you feedback for what kind of mental moves might be better and why.[1]

E.g. for learning programming, learning to solve math problems, learning to write emails well, ...

Rich

When someone tells you “you screwed up”, this is less useful than when someone tells you “you screwed up, here’s what you should’ve done instead …” (assuming accurate feedback). Supervised learning is faster than reinforcement learning.

Example Applications:

After you solved a problem, review what the key insights were, and if possible also how you found them, and also where you wasted time on unproductive angles. Ask yourself how you could’ve succeeded faster. What were the key properties of the thinking avenues which were or were not us

...
outlining is a historically recent underutilized gift to family
by daijin — 2025-02-26T13:58:17.623Z

outlining is specialized work which reduces a text to complete summary statements and collapsed detail.

an outline containing a work sprint. note the collapsed points in the 'old sprints' which hide all the old sprint detail.

outlining is historically recent, since particular digital interfaces (such as Workflowy, Org Mode, Dynalist or Ravel) make it orders of magnitude easier to reorganize and nest text.

good outlining is an act of service towards readers in an ever-more-complex world.

(family = aligned successors, in general.)

outlining is specialized work which reduces a text to complete summary statements and collapsed detail.

outlining includes hiding theatre from a work, and grouping, prioitizing and subcategorizing its statements, and results in a set of outline-headings like the ones for this article.

an efficient outline hides or eliminates the theatre from a work. theatre includes pathos, ethos, logos, and more. theatre is to writing is as populism to democracy: a piece of writing often has a bit of theatre to keep the reader (and writer) incentivized to continue, but too much theatre and you end up with nietzsche's later works where theatre takes place of substance and everything becomes completely unintelligible.

outlining is antithetical to stream-of-consciousness because of grouping, prioritization and subcategorization. this is orthogonal to writing and speaking, despite the pre-digital notion of writing being more formal than printing, which had worked because printing required far more resources than typing does today. today, written works may not have their statements properly grouped, prioritized and subcategorized, as writing has become cheap as words. an outline is easy to distinguish from a wall of text (it is significantly shorter), and a good outline can be recursively determined to verify its quality of grouping, prioritization and subcategorization.

an outline-made-of-headings is distinct from theatre-headings as typically used in books, posts or journal papers. the outline of this work is exactly as you read on the sidebar. each heading is a complete statement; reading the outline gives you a summary of the text. I could have instead made headings like "what is outlining?" "why is outlining historically recent?" "why is outlining important for society?", which do not give you the main point straight away but instead aim to compe...
Osaka
by lsusr — 2025-02-26T13:50:24.102Z

The more I learn about urban planning, the more I realize that the American city I live in is dystopic. I'm referring specifically to urban planning, and I'm not being hyperbolic. Have you ever watched the teen dystopia movie Divergent? The whole city is perfectly walkable (or parkourable, if you're Dauntless). I don't know if it even has cars. The USA's urban planning is so bad it's worse than a literal sci-fi dystopia.

[One of the above images is a screenshot from the young adult dystopia movie Divergent. The other is a photo of Tacoma, Washington.]

Sometimes I consider moving to a city with more weirdo nerds. Then I remember Lightcone's headquarters is in San Francisco. On the cost of living vs good urban planning tradeoff, "Let's all move to San Francisco" is the worst coordination failure I can imagine.

It is said that urbanists are radicalized by Amsterdam or Tokyo. Well, I'm in Osaka right now, and as far as I can tell, it's just as good as Tokyo. If you want to know why Japanese cities are so awesome, go watch this video by Not Just Bikes which undersells how good Japan's cities are, because the video discusses only the streets, without mentioning how many small businesses it is possible to pack into a building. These effects are multiplicative. There are about as many coffee shops within walking distances of my random hotel in downtown Osaka than there are anywhere in Seattle, the city Starbucks originated in. This isn't because Osaka is especially into coffee. It's because Osaka has such high density and walkability that there's a huge quantity of all businesses within walking distance, some of which happen to be coffee shops—and that's before considering Osaka's excellent subway system. Restaurants aren't just convenient. They're cheap too, especially if you earn income in USD.

It's not perfect. Most importantly to me, the chairs are all for Japanese-sized people. I'm taller than that.

I notice that high-agency people are very likely to move cities. Usually this is for work or entrepreneurship. But if you have the freedom to choose what city you live in, then it may be worth optimizing for quality of life.