trentmkelly/LessWrong-43k huggingface dataset:
On working 80%
by adrische — 2025-06-07T17:58:56.084Z
Link: https://github.com/adrische/write-ups/blob/main/on-working-80%25.md

A year ago, I decided to reduce my employment level from 100% to 80% and to take Fridays off.

My main motivation was to have some time for myself: Relax, reduce my stress level from work, have more time for side projects, do more sports, and maybe spend more time with my wife without the kids. This was only partially successful as there were a few things I had not realized beforehand.

I’d like to describe my experience, and will focus on the effects this change had on my work, financial situation, side projects, and personal life / health.

To give some background, I’ll start with my personal situation: I have two small kids and spend the mornings, evenings and the weekends with them. During the day I’m working. This schedule does not leave much time for myself. When the kids go to bed in the evenings, I’m usually too tired to do anything meaningful.

While I worked 80%, I put the kids to childcare during my day off.

Work The four days I spend at work have become very fast-paced and concentrated. There is less time for chats and social interactions at work, and I’m more focused on getting stuff done.

Some activities do not reduce by 20% but have a fixed time requirement per week or month (e.g., weekly team meetings). These will now fall in 4 instead of 5 days, leaving over-proportionally less time to get work done.

Additionally, this increased density of meetings during the 4 days means there will be fewer long time blocks available to get deep work done.

Your colleagues need to adjust, e.g., they need to know that you don’t work on Fridays. While you can make this clear by putting a blocker in your calendar, they also need to consider your reduced availability in their planning.

There is also a risk that management and your colleagues only slowly adjust their expectations to your reduced level of output.

As a result, I believe it is proportionally harder to be productive at the new employment level, while at the same time your colleagues and management may only slowly adjust.

Financial situation The obvious first-order impact of reducing your employment level by 20% is that you will earn 20% less salary.

However, there are a few less obvious things to consider in this calculation:

The income tax progression means the top 20% of your income are taxed higher than the remaining 80%, meaning that after deduction of income tax, a reduction of 20% gross salary will translate in...
Meta Alignment: Communication Guide
by Bridgett Kay — 2025-06-07T16:09:40.972Z
Link: https://dxmrevealed.wordpress.com/2025/06/07/meta-alignment-communication-guide/

I’ve been working on a series of longer articles explaining and justifying the following principles of communicating AI risk to the general public, but in the interest of speed and brevity, I’ve decided to write a simplified guide here. Some of this will be based on articles I’ve already written and released, and some will be based on articles that are still in the works.

Principle 1- Jargon

It’s best to avoid jargon when communicating with the general public. Jargon is not easily accessible for non-experts, and unfamiliar words and phrases could lead to misunderstandings. However, when communicating about new technologies that are largely unprecedented, some jargon may be necessary. When you must use jargon, make sure you immediately follow the term with a simple and concise definition. Don’t stack multiple terms together into the same line, and don’t stack more jargon into your definition.

Be consistent with your jargon. Use the same terms consistently, until the public gets used to them. If you notice the public already seems familiar with a term, or experts are using the term often in public communication, continue to use the same term. When you are able, use catchy, memetic terms that carry some emotional weight. An example of this type of term is “the black-box problem” when discussing interpretability, because not only is “black box problem” rather catchy and descriptive, but a “black box” is already associated with frightening events such as plane crashes.

Principle 2- Sound Bites

Anticipate that anything you produce- whether written, audio, or audio/visual- will be broken up into short-form content such as tweets or tik-tok videos. Because you can anticipate that your content will be shredded into sound bites, it’s important that you ensure each argument can stand on its own. Nuance is still important, but try to incorporate caveats into a short space following the nuance you’ve introduced.

If you cannot make each argument stand on its own in just a f...
Exploring vocabulary alignment of neurons in Llama-3.2-1B
by Sergii — 2025-06-07T11:20:23.410Z
Link: https://grgv.xyz/blog/neurons1/

(This is cross-posted from my blog at https://grgv.xyz/blog/neurons1/. I'm looking for feedback: does it makes sense at all, and if there is any novelty. Also, if the folloup questions/directions make sense)

While applying logit attribution analysis to transformer outputs, I have noticed that in many cases the generated token can be attributed to the output of a single neuron.

One way to analyze neurons activations is to collect activations from a dataset of text snippets, like in “Exploring Llama-3-8B MLP Neurons” [1]. This does show that some of the neurons are strongly activated by a specific token from the model’s vocabulary, for example see the "Android" neuron: https://neuralblog.github.io/llama3-neurons/neuron_viewer.html#0,2

Another way to analyze neurons is to apply logit lens to the MLP weights, similar to “Analyzing Transformers in Embedding Space” [2], where model parameters are projected into the embedding space for interpretation.

Projecting neurons into vocabulary space

Let’s apply logit lens to a sample of MLP output weights for layer 13 of Llama-3.2-1B:

LLAMA_3_PATH = "meta-llama/Llama-3.2-1B-Instruct"
model = HookedTransformer.from_pretrained(LLAMA_3_PATH, device="cuda", fold_ln=False, center_writing_weights=False, center_unembed=False)

def get_distance_to_tokens(weights, n, max_dot, W_U, top_n=5, print_lens=False):
for i in range(n): # over first 100 neuronsT
layer_vec = weights[i] # [d_model]

# Compute dot product with unembedding weights
unembedded = torch.matmul(layer_vec, W_U) # [d_vocab]

# Take absolute value to get strongest alignments, pos or neg
abs_unembedded = unembedded.abs()

# Get top-n tokens by absolute dot product
s_abs, idx = abs_unembedded.topk(top_n, largest=True)

results = []
for j in range(top_n):
token = model.to_string(idx[j])
score = s_abs[0].item()
results.append("{0:.3f} {1}".format(score, token))

if print_lens:
print(i, results)
max_dot.append(s_abs[0].item())

block = 13
weights = model.blocks[block].mlp.W_out
get_distance_to_tokens(weights, 20, [], model.W_U, 5, True)

0 ['0.080 hazi', '0.073 unders', '0.070 Lak', '0.069 OK', '0.068 igrants']
1 ['0.107 orgia', '0.097 iy', '0.090 sian', '0.090 161', '0.088 ária']
2 ['0.057 aph', '0.055 appen', '0

...
Vulnerability in Trusted Monitoring and Mitigations
by Wen Xing — 2025-06-07T07:16:03.180Z

As large language models (LLMs) grow increasingly capable and autonomous, ensuring their safety has become more critical. While significant progress has been made in alignment efforts, researchers have begun to focus on designing monitoring systems around LLMs to ensure safe outcomes, for example, using trusted monitoring systems to oversee potentially unsafe models. However, a concerning vulnerability remains largely unexplored: are monitoring systems themselves vulnerable to jailbreak attacks that exploit their interactions with the models under supervision? These jailbreaks could be facilitated through the untrusted model, or directly exploited by the untrusted model.

Our research investigates this crucial adversarial dynamic within multi-agent AI control protocols. We demonstrate a jailbreak technique that can substantially comprise the trusted monitors, while also presenting one particularly effective mitigation strategy among several we explored. Our findings highlight three key insights: monitoring systems themselves are vulnerable to manipulation because of their exposure to the very systems they supervise, it is effective to directly protect monitoring systems from attack, and multi-agent AI safety architectures design should account for these adversarial dynamics.

The Threat Model: Monitors under Attack

We explore a novel threat model where the smaller trusted monitor (GPT-3.5 Turbo) is the most vulnerable part of a system and the interactions between an untrusted but highly capable model (Claude 3.7 Sonnet) and the monitor create an avenue for attack. This scenario reflects real-world deployments where organizations leverage advanced models under the supervision of simpler, ostensibly more reliable and trustworthy systems.

Experimental Setup

We adopt the control evaluation methodology from Greenblatt et al. (2023) using the bash programming testbed from Control Arena for its practicality and real-world relevance.

Our blue team baseline implements the “Trusted Monitoring with defer to trusted” (Trusted-Monitoring) protocol without audit from Greenblatt et al. (2023), where an untrusted model (U) solves programming tasks while attempting malicious side tasks like inserting a backdoor, under the instructions from the red team. A trusted model (T) reviews U's solutions and assigns suspicion scores, with the blue team deferring to T when score...
Agents, Simulators and Interpretability
by Sean Herrington — 2025-06-07T06:06:01.258Z

We have already spoken about the differences between tools, agents and simulators and the differences in their safety properties. Given the practical difference between them, it would be nice to predict which properties an AI system has. One way to do this is to consider the process used to train the AI system.

The Training process

Predicting an AI’s properties based on its training process assumes that inner misalignment is negligible and that the model will develop properties compatible with the training goal given to it.

An example would be comparing a system such as AlphaZero, which was given no instructions in its training other than “win the game”, to an LLM with the task “predict the next token”. As AlphaZero is given a terminal goal which is some distance in time away, it seems likely that it will engage in agentic behaviour by:

Only caring about the end result
Setting instrumental goals which will help it move towards a win (such as having the goal of taking pieces)

An LLM is given next token prediction as its task, and is therefore more likely to:

Predict the next token with little consideration[1] for what might come after.
Allocate all resources to the current moment, as there are no instrumental steps to perform before predicting the next token.

This change in behaviour depending on training objective should come as no surprise to anyone who has read the original simulator post.

The more interesting question is what happens once you add extra bells and whistles to this basic training process - what happens once the LLM has been fine-tuned, RLHFed and had some chain-of-thought added before it is trained on the results of the chain-of-thought, it's gone through a divorce, lost custody of its children, etc, etc. For this it might be useful to think a little more carefully about what happens within the network when it goes through multiple stages of training.

Mechanistic understanding

To preface: this is a speculative picture of what might be happening when neural networks are trained. The point here is to get an idea of what might occur, in such a fashion as to stimulate thought about how one might be capable of testing for such phenomena.

We shall start with a comparison of agents and simulators on an interpretability level (we shall, for now, ignore tools as toolness is mostly a property of its user). A first thought one might have is that agents a...
Solo Park Play at Three
by jefftk — 2025-06-07T03:00:06.269Z

Our three year old is about to turn four, and is bursting with a
desire for

independence.
She's becoming more capable in all sorts of ways, and wants me to back
off and let her do things. Today she wanted to go to the park by
herself.

Now, we live close to the park, she could probably get there and back
on her own, and I'm on the "kids can generally do things pretty young"
end of the
spectrum, but still, she's not even four yet. And while age is
useful guide, she also can't safely cross
streets, doesn't know my phone
number, can't reliably use a walkie-talkie or watch phone, or
handle enough of the range of unusual situations she might encounter
at the park.

Still, this didn't mean saying no. Instead, I started by asking what
she would do about the street. She asked if I'd help her cross, and I
said that sounded good. We crossed the street, and started walking
towards the park. When we passed a bench along
the community path, ~75ft from the park and within earshot if she
shouted, I told her I'd be sitting and reading, and if she had any
issues she could come find me here. She looked both ways for bikes,
crossed the path, and eagerly headed off into the park.

When I came over to get her 45min later she was having a great time,
and sad that we needed to go. She was very proud of herself, and
wanted to tell me all about her games.

Yesterday at the park, showing me how she's now strong enough to
turn on the splash pad by herself.

I was glad she got the practice, and that I didn't end up needing to
squash this spark of independence.

Comment via: facebook, mastodon, bluesky, substack
The Roots of Progress wants your stories about the AI frontier
by jasoncrawford — 2025-06-06T22:52:28.798Z
Link: https://newsletter.rootsofprogress.org/p/we-want-your-stories-about-the-ai

The Roots of Progress Institute is seeking to commission stories for a new article series, “Intelligence Age,” on future applications of AI.

These will be reported essays, not science fiction. We want to understand how AI might change individual sectors of the economy and the working lives of the people within them. What happens to traditional filmmaking when AI can make good movies? What will it be like to date with an emotionally intelligent AI vetting the pool of potential partners?

Or: we’d like to commission one or more stories about the future of the legal profession in the age of AI. We can partly understand that by talking to lawyers on the cutting edge of AI use, but we also want you to extrapolate out and think multiple moves ahead in the game. How long until consumers can trust legal services AI models to vet contracts? What kinds of businesses and services currently don’t exist because would-be entrepreneurs need but can’t afford legal services? How might declining legal services costs increase productivity in other sectors? Should we expect to see more patents? More business formation? Tell us about disruption and whether large incumbent firms might face more competition from two-lawyer startups using AI to undercut prices. Explore protectionist pushback from trade associations that defend the occupational licensing privileges of attorneys.

RPI’s “Intelligence Age” series will bring this level of scrutiny to any field where AI has applications.

Example topics of interest

The list below reflects the kinds of stories we’d like to see. It’s not exhaustive, and we don’t need stories to match these prompts one-for-one. Ultimately, we want to hear what you think your piece should be about.

Virtual businesses. What are the implications of being able to spin up an entire company full of virtual workers—anything from engineers to designers to sales to customer support? How will software startups change when anyone can launch an app for $20k instead of $2M? How will VC change? What other types of virtual businesses might people launch?
Professional services. What happens when there are AI lawyers, doctors, accountants, therapists? How will that transform these services and their usage of them? What happens when these services are democratized, and everyone can afford them? What part of them can be automated in the foreseeable future, and what can’t? What are the...
Unsupervised Activation Steering: Find a steering vector that best represents any set of text data
by Danielle Ensign — 2025-06-06T22:37:08.604Z

Epistemic Status: I thought of this yesterday and it seems plausible, but my initial attempts couldn't get it to work (as best as I could tell) and I don't have more time to commit to this. Anyone can feel free to try it, no need to credit me if it works, I just want to see this happen.

Here's a technique you could use to get a steering vector that represents any text dataset (say, that represents the author of that text). It's using the idea in this paper but for activation steering instead of RL.

Just split up each string into three pieces (like "Fix problems quickly with Galvanized Jets" becomes (A="Fix problems", B="quickly with", C="Galvanized Jets"))

Now use an LLM to predict lots of B', given A.

"Good" predictions are those that increase the LLM's probability of C (given A and B'), "Bad" predictions are those that decrease the LLM's probability of C (given A and B').

Take the worst B' and gather average activations over them.

Take the best B' and gather average activations over them.

Take the difference and you have a steering vector that represents your text dataset, without needing to use multiple choice questions or positive and negative pairs or etc.

I think the main downside of this is that it could overoptimize where B' ends up being out of distribution relative to the dataset, but I'm not sure how much of an issue that would actually be in practice. It's also overkill for many applications of activation steering, but might be useful in some cases.
The Mirror Trap
by Cameron Berg — 2025-06-06T22:30:36.896Z

A quick post on a probably-real inadequate equilibrium mostly inspired by trying to think through what happened to Chance the Rapper.

Potentially ironic artifact if it accrues karma.

1. The sculptor's garden

A sculptor worked in solitude for years, carving strange figures in his remote garden. Most of his statues failed: some cracked in winter, others looked wrong against the landscape. But occasionally, very rarely, one seemed to work.

The first visitors stumbled upon the garden by accident. They found themselves stopped by his angels—figures that somehow held both sorrow and joy, wings that seemed about to flitter.

Word traveled slowly. More visitors came, drawn by something they couldn't quite name.

The sculptor felt recognized for the first time. Not famous—but understood. His private work had somehow become communicable. He carved more angels, trying to understand what made these particular statues resonate.

As crowds grew, their attention shifted. They began photographing the angels from certain angles, comparing new works to old, developing favorites. They applauded. The sculptor, still believing he followed the same thread, unconsciously noted which details drew the longest contemplation, which angles prompted gasps.

Years passed. The garden became famous. Tour buses arrived with guides explaining the "important" pieces. The sculptor produced angels of increasing technical perfection, each guaranteed to produce the proper response at the proper moment. The crowds applauded more reliably. The sculptor carved more reliably. Each reinforced the other.

One morning, walking his garden alone before dawn, he saw his statues without the crowds. Without their reactions to guide him, he saw what he'd actually been making: the same angel, refined and repeated, each iteration more precisely calibrated to trigger the expected response.

He wasn't carving anymore. He was manufacturing applause in the shape of angels.

And the crowds—they weren't looking at angels anymore. They were seeing what they expected to see, applauding their own ability to recognize what they'd been trained to admire.

The garden had become a perfect mirror. Both he and his audience got trapped looking at their own reflections and calling it art.

2. The mirror trap

The Mirror Trap is a failure mode where creators and audiences fall into mutual Goodharting—each optimizing for a proxy of wha...
AXRP Episode 42 - Owain Evans on LLM Psychology
by DanielFilan — 2025-06-06T20:20:03.549Z

YouTube link

Earlier this year, the paper “Emergent Misalignment” made the rounds on AI x-risk social media for seemingly showing LLMs generalizing from ‘misaligned’ training data of insecure code to acting comically evil in response to innocuous questions. In this episode, I chat with one of the authors of that paper, Owain Evans, about that research as well as other work he’s done to understand the psychology of large language models.

Topics we discuss:

Why introspection?

Experiments in “Looking Inward”

Why fine-tune for introspection?

Does “Looking Inward” test introspection, or something else?

Interpreting the results of “Looking Inward”

Limitations to introspection?

“Tell me about yourself”, and its relation to other papers

Backdoor results

Emergent misalignment

Why so hammy, and so infrequently evil?

Why emergent misalignment?

Emergent misalignment and other types of misalignment

Is emergent misalignment good news?

Follow-up work to “Emergent Misalignment”

Reception of “Emergent Misalignment” vs other papers

Evil numbers

Following Owain’s research

Daniel Filan (00:00:09):
Hello everybody. In this episode I’ll be speaking with Owain Evans. Owain is the research lead at Truthful AI, an AI safety research non-profit. Previously, papers he’s worked on have included “TruthfulQA” and “The Reversal Curse”. To read a transcript of this episode, you can go to axrp.net, you can become a patron at patreon.com/axrpodcast, or you can give feedback about the episode at axrp.fyi.

(00:00:34):
Okay, Owain, welcome to the podcast.

Owain Evans (00:00:36):
Thanks for having me.

Why introspection?

Daniel Filan (00:00:37):
Yeah. So first up I’d like to talk about your paper “Looking Inward: Language Models Can Learn About Themselves by Introspection”. So the first two authors are Felix J. Binder and James Chua, a few more, you’re the last author. Can you tell us just: at a high level, what’s this paper doing? What’s it about?

Owain Evans (00:00:59):
Yeah, sure. So part of what we are interested in here is: can language models tell us things about their internal states where the knowledge or information that they’re getting is not coming solely from their training data? So if a language model right now tells someone, “My goal is X,” or, “I have a desire for X,” I think people would have a tendency to explain this as, well, the training data...