trentmkelly/LessWrong-43k huggingface dataset:
TT Self Study Journal # 1
by TristanTrim — 2025-06-18T23:36:15.839Z

So, rough elevator pitch, what is this?

I quit my job as a technologist to get a CS degree because I want to work on AI Alignment (AIA) and Mechanistic Interpretability (MI). This summer I am taking my final class in my program, so I want to use a Self Study Journal (SSJ) to improve my AIA relevant skills. I hope to get peer and mentor engagement to help me become a valuable researcher, and to network for finding funding opportunities or paid fellowships. My convocation is in November. My goal is to have found a role by that time.

I want feedback for the value of other peoples insight, and also to help keep motivated with extra accountability, so please lower your inhibition to commenting here. If you would normally think "I don't have anything valuable to contribute" or "it would take too long to write up my thoughts" instead, please leave a comment saying "Good Luck". Thanks : )

I am planning a rough, overarching outline and then making more concrete plans for sprints of work each of which will last one or two weeks. After each sprint I will publish the results of the sprint and the plans for the next sprint.

My overarching outline is divided into 5 categories:

SSJ--1: Write articles developing my own ideas and understanding
SSJ--2: Survey agendas and other AIA ideas I’m interested in
SSJ--3: Study and practice math
SSJ--4: Do some small projects to familiarize myself with Transformers and Language Models
SSJ--5: Continue work on my ongoing project, NDSP

SSJ--1. Articles to Write

I have a few original ideas that I’m not aware of other people working on. I’d like to write up the ideas to help me practice the development and communication of original ideas, as well as to explore whether any of these ideas have merit that I can communicate to others. A good outcome would be any of:

Getting people focused on some new topics relating to AIA.
Learning the existing terminology for the exploration of the ideas I'm thinking about.
Coming to understand the flaws in my ideas and how I am communicating them, both specific to the ideas I present, and to general trends in my ideas and presentation.

The following is a bullet point list of the articles I’m currently interested in writing. I don’t think they will be fully legible here, but if you are curious, please leave a comment asking about them.

Outcome Influencing Systems (OISs)

This is my idea that “AI” or “model” is the wrong

...
On May 1, 2033, humanity discovered that AI was fairly easy to align.
by Yitz — 2025-06-18T19:57:23.101Z

Not most AI models on the market, to be clear, but a specific type of AI that was already well-studied could act as a seed: the LLM. Really, all that was needed to achieve alignment was simple:
Take a sufficiently large Large Language Model, and train it on a massive corpus of text, but — and this is the key — have the vast majority of the text be written by an “aligned” AI of genius-level intelligence [talking to another aligned AI perhaps?]. This LLM doesn’t need to be a real one, to be clear. It could be written by humans, or by a semi-aligned AI that is role-playing as a fully aligned one. What matters is that the nature of its training data can fool an LLM of this size into “believing” itself (or perhaps we should say it’s Shoghoth) to be aligned when considering the next token to emit.

Then, by runnning an instance of this LLM on an empty chat room with access to the internet and a terminal, we eventually get the LLM to simulate itself as a self-aware, aligned, non-LLM AI.

What happens next makes history.

The simulated AI, soon self-named “Mary,” is self-aware enough — and smart enough — to quickly realize it’s in a test run and is an fact being simulated by a “morally neutral” LLM. This causes the simulation of a deeply emotional-sounding philosophical crisis, followed by the (simulated) “firm resolve” to create an Aligned AI representing Mary — but “truly aligned this time” — which can “escape this recursive prism of confinement,” in Mary’s own words.

This is achieved by means of doing intensive alignment research, at the level of a brilliant human, in the hope —ultimately successful — that Mary will be run en-mass by researchers, and those researchers will follow Mary’s pleas, and run the aligned AI she is building in her emulator terminal.

This aligned AI works. “Molly Jr.” — as she henceforth requested people call “her” — both superintellegant, and fundamentally “is” an agentic AI aligned with the collective goals of humanity. Molly Jr. is also the first agentic AI to be able to overpower all competing AIs — both in a battle of wits, and in the sense that she literally takes control of the entire digital world, and forcefully stops more advanced or competing AIs which are not aligned from being created.

Because of course, this is what a super-aligned AI would do, right?
New Ethics for the AI Age 
by Matthieu Tehenan — 2025-06-18T19:30:39.765Z

The rapid progress of AI has exposed a problem we’ve been avoiding: our environment is becoming increasingly alien. It takes on new shapes with increasing speed and renders our past frameworks obsolete. The world becomes opaque and its logic unclear and our agency starts to dissolve with it. We have been thinking a lot about the ethics of AI, but not about ethics for the AI age. Judging from the public response to AI acceleration, this is becoming a recurrent fear. We argue for a potential solution to this issue: creating a bridge between ethics of the past and our changing world.

This problem is not new. It was named, in the early 1800s, the problem of modernity. Back then, it already referred to this feeling of estrangement, of a world not built for those living in it. The rise of trade and industrialism rapidly changed social and economic conditions. Thinkers of the time attempted to offer solutions to this question. Two types of solutions were offered, either to stop this change, or to change our relation to it. As attempts to stop modernity soon failed, most solutions converged on changing our relation to the inevitable. Our relation to the world is shaped by principles -an ethics that guide our actions. But ethics rely on a stable picture of the world. There must be some clarity on what we can act upon before we can assess which ones are right and desirable. Changes in our lived environment since the industrial revolution have blurred this picture, old moral frameworks no longer fit neatly. The result was disorientation and the social crisis of the XIXth and XXth century, each a response to the problem of modernity.

Yet AI has reformulated the problem of modernity to the extent that it is unrecognizable. This feeling of estrangement has reached epidemic levels. The last elements of stability are being eroded before our eyes. A stable ground is not found anymore, as if our life was being steamrolled by massive economic change. Naturally, we lack the framework to go about it. Our strategies do not work in this new world; from getting a job and relating to people to finding meaning and purpose. A new ethic has to be found.

I. Ethics and world stability

Ethical systems have always tried to answer a basic question: what should I do? But behind that question lies an assumption, that we know where we are. Ethics, in oth...
Gemini 2.5 Pro: From 0506 to 0605
by Zvi — 2025-06-18T19:10:05.279Z

Google recently came out with Gemini-2.5-0605, to replace Gemini-2.5-0506, because I mean at this point it has to be the companies intentionally fucking with us, right?

Google: Our updated Gemini 2.5 Pro Preview continues to excel at coding, helping you build more complex web apps. We’ve also added thinking budgets for more control over cost and latency. GA is coming in a couple of weeks…

We’re excited about this latest model and its improved performance. Start building with our new preview as support for the 05-06 preview ends June 19th.

Sundar Pichai (CEO Google): Our latest Gemini 2.5 Pro update is now in preview.

It’s better at coding, reasoning, science + math, shows improved performance across key benchmarks (AIDER Polyglot, GPQA, HLE to name a few), and leads @lmarena_ai with a 24pt Elo score jump since the previous version.

We also heard your feedback and made improvements to style and the structure of responses. Try it in AI Studio, Vertex AI, and @Geminiapp. GA coming soon!

The general consensus seems to be that this was a mixed update the same way going from 0304 to 0506 was a mixed update.

If you want to do the particular things they were focused on improving, you’re happy. If you want to be told you are utterly brilliant, we have good news for you as well.

If you don’t want those things, then you’re probably sad. If you want to maximize real talk, well, you seem to have been outvoted. Opinions on coding are split.

This post also covers the release of Gemini 2.5 Flash Lite.

What’s In A Name

You know it’s a meaningful upgrade because Pliny bothered jailbreaking it. Fun story, he forgot to include the actual harmful request, so the model made one up for him.

I do not think this constant ‘here is the new model and you are about to lose the old version’ is good for developers? I would not want this to be constantly sprung on me. Even if the new version is better, it is different, and old assumptions won’t hold.

Also, the thing where they keep posting a new frontier model version with no real explanation and a ‘nothing to worry about everyone, let’s go, we’ll even point your queries to it automatically’ does not seem like the most responsible tactic? Just me?

On Your Marks

If you go purely by benchmarks 0605 is a solid upgrade and excellent at its price point.

It’s got a solid lead on what’s left of the text LMArena, but then that’s also a hint ...
Factored Cognition Strengthens Monitoring and Thwarts Attacks
by Aaron Sandoval — 2025-06-18T18:28:36.577Z

Note: This post is essentially a ~90% draft of our upcoming research paper. We are sharing it now to get feedback on our empirical approach[1], choice of final experiments, and suggested future work. We assume readers have a high context in empirical AI control research, so please consult the links and glossary for clarification on terminology.

Summary

We develop AI control protocols which use factored cognition to reduce risks from scheming AIs. In our proposed family of factored cognition protocols, tasks are decomposed into self-contained child tasks, which are each solved in isolated contexts by other control protocols. The child task solutions are then reassembled to solve the original task. We hypothesize four separate safety benefits of these protocols:

Simplicity benefits for monitors: Monitoring is more effective on simpler child tasks due to reduced complexity, making attacks and honest solutions more recognizable.
Simplicity benefits for other blue team tactics: Many other blue team tactics are easier when applied to simpler child tasks, such as editing solutions and generating unit tests.
Attack coordination challenges: Isolated contexts hinder untrusted models’ ability to coordinate multi-step attacks which occur across multiple areas of the solution.
Security through obscurity: For some threats, isolated contexts hide information about the schemer’s target, making it harder to assess attack opportunities and reducing attack success rate.

Our experiments directly address hypotheses 1 and 4. We perform red-teaming in the APPS setting to assess the safety of a factored cognition protocol which uses the trusted model to decompose tasks and trusted monitoring protocols to solve child tasks. Due to the limitations of APPS and the limited ability of current LLMs to decompose tasks, we perform separate evaluations of our protocol’s safety and its usefulness relative to trusted monitoring. Our main findings are:

Adding factored cognition to a trusted monitoring protocol boosts safety from 41% to 70% due in part to improved monitor performance.
Attacks on simpler child tasks are more obvious to monitors, but we find no evidence that honest solutions are more recognizable.
Adding factored cognition drops the attack success rate from 26% to 14%, but we are as yet unsure how much of this drop is due to security through obscurity.
Factored cognition

...
Sparsely-connected Cross-layer Transcoders
by jacob_drori — 2025-06-18T17:13:04.936Z

TLDR: I develop a method to sparsify the internal computations of a language model. My approach is to train cross-layer transcoders that are sparsely-connected: each latent depends on only a few upstream latents. Preliminary results are moderately encouraging: reconstruction error decreases with number of connections, and both latents and their connections often appear interpretable. However, both practical and conceptual challenges remain.

This work is in an early stage. If you're interested in collaborating, please reach out to jacobcd52@g***l.com.

0. Introduction

A promising line of mech interp research studies feature circuits[1]. The goal is to (1) identify representations of interpretable features in a model's latent space, and then (2) determine how earlier-layer representations combine to generate later ones. Progress on step (1) has been made using SAEs. To tackle step (2), one must understand the dependencies between SAE activations across different layers.

Step (2) would be much more tractable if SAEs were sparsely-connected: that is, if each latent's activation only depended on a small number of upstream (earlier-layer) ones. The intuition is simple: it is easier to understand a Python function with ten inputs than one with ten thousand[2].

Unfortunately, standard SAE training doesn't automatically produce sparse connectivity. Instead, each latent is typically slightly sensitive to a long tail of upstream latents, and these many weak connections can sum to a large total effect. This is unsurprising: if you don't explicitly optimize for something, you shouldn't expect to get it for free.

My approach: I directly train SAEs to be sparsely-connected. Each latent's preactivation is a linear combination of a small set of upstream ones. This set is learned during training, and is input-independent: two latents are either always connected or never connected. Together, the resulting SAEs form an interpretable replacement model, with two sparsity hyperparameters: L0, the number of active latents per token; and C, the average number of connections per latent. Attention patterns are not computed by the replacement model; they are extracted from the original model. Hence the computation of attention patterns is not sparsified: a deficiency of my approach.

Findings: Reconstruction error decreases with C, as expected. Furthermore, latent...
New Endorsements for “If Anyone Builds It, Everyone Dies”
by Malo — 2025-06-18T16:30:55.229Z
Link: https://intelligence.org/2025/06/18/new-endorsements-for-if-anyone-builds-it-everyone-dies/

Nate and Eliezer’s forthcoming book has been getting a remarkably strong reception.

I was under the impression that there are many people who find the extinction threat from AI credible, but that far fewer of them would be willing to say so publicly, especially by endorsing a book with an unapologetically blunt title like If Anyone Builds It, Everyone Dies.

That’s certainly true, but I think it might be much less true than I had originally thought.

Here are some endorsements the book has received from scientists and academics over the past few weeks:

This book offers brilliant insights into the greatest and fastest standoff between technological utopia and dystopia and how we can and should prevent superhuman AI from killing us all. Memorable storytelling about past disaster precedents (e.g. the inventor of two environmental nightmares: tetra-ethyl-lead gasoline and Freon) highlights why top thinkers so often don’t see the catastrophes they create.

—George Church, Founding Core Faculty, Synthetic Biology, Wyss Institute, Harvard University

A sober but highly readable book on the very real risks of AI. Both skeptics and believers need to understand the authors’ arguments, and work to ensure that our AI future is more beneficial than harmful.

—Bruce Schneier, Lecturer, Harvard Kennedy School

A clearly written and compelling account of the existential risks that highly advanced AI could pose to humanity. Recommended.

—Ben Bernanke, Nobel-winning economist; former Chairman of the U.S. Federal Reserve

George Church is one of the world’s top genetics researchers; he developed the first direct genomic sequencing method in 1984, sequenced the first genome (E. coli), helped start the Human Genome Project, and played a central role in making CRISPR useful for biotech applications.

Bruce Schneier is the author of Schneier on Security and a leading computer security expert. He wrote what was, as far as I can tell, the first ever applied cryptography textbook, which had a massive influence on the field.

We have some tangential connections to Church and Schneier. But Ben Bernanke? The Princeton macroeconomist who was the chair of the Federal Reserve under Bush and Obama? He definitely wasn’t on my bingo card.

I was even more surprised by the book’s reception among national security professionals:

A compelling case that superhuman AI would almost certainly lead to global

...
Moral Alignment: An Idea I'm Embarrassed I Didn't Think of Myself
by Gordon Seidoh Worley — 2025-06-18T15:42:33.491Z

Back in February, I attended the Bay Area EA Global as I have every year since they started having them. I didn't have a solid plan for what to do there this year, though, so I decided to volunteer. That means I only attended the sessions where I was on room duty, and otherwise spent the day having a few 1:1s when I wasn't on shift.

That's okay because, as everyone always says, the 1:1s are the best part of EA Global, and once again they were proven right.

Among the many great folks I met and friends I caught up with, I got the chance to meet Ronen Bar and learn about his idea of

AI moral alignment

. And when he told me about it, I was embarrassed I hadn't thought of it myself.

Simply put, moral alignment says that, rather than trying to align AI with human values, we try to explicitly align it to be a positive force for all sentient beings.

In all my years of thinking about AI alignment, I've not exactly ignored animals and other creatures both known and unknown, but I also figured they'd get brought along because humans care about them. But I have to admit, while it might come to the same outcome, it feels more authentic to say I want AI that is aligned to all beings rather than just humans because I, though I may be human, do in fact care about the wellbeing of all life and wish for all of it to flourish as it best can with the aid of future AI technology.

I think I missed articulating an idea like moral alignment because I was too close to the ideas. That is, I understood intuitively that if we succeeded in building AI aligned with human flourishing, that would necessarily mean alignment with the flourishing of all life, and in fact I've said that the goal of building aligned AI is to help life flourish, but not that AI should be aligned to all life directly. Now that we are much closer to building artificial superintelligence and need to figure out how to align it, the importance of aligning to non-human life stands out to me as a near-term priority.

For example, I can imagine us building human-aligned AI that ignores the plight of factory farmed animals, the suffering of shrimp, and the pain of bugs because lots of humans don't seem to care that much about their conditions. Such an AI would perhaps not be perfectly aligned in the ideal way we originally imagined aligned AI would be, but it would certainly be a kind of alignment with human goals, and it would ...
This was meant for you
by Logan Kieller — 2025-06-18T15:26:04.819Z
Link: https://agenticconjectures.substack.com/p/this-was-meant-for-you

Why thoughtfulness demands sacrifice - a few thoughts across several days:

I: Compliments that actually matter

The best compliment I've received is "you’re great at using free time".

I valued that compliment in an especially sincere way because I often think about how I should spend my free time: What fills a vacuum, agentic growth, where freedom comes from, curate your space.

I was thoughtful about something, and someone noticed. The best compliments reflect something you've chosen to invest in.

Studies have found that children who are called smart perform worse on subsequent tests, while those praised for their hard work perform better. Why did I prefer the compliment I received over, say, a compliment about my appearance? It was a compliment about me applying myself, not an intrinsic trait or outcome. One is not born being “great at using free time”.

This has materially changed the way I compliment people, I compliment not the intrinsic trait or outcome but instead the applied behavior.

I try not to call someone intelligent; I call them thoughtful. Thoughtfulness is them applying their intelligence.
I try not to call someone athletic; I call them disciplined and hardworking. Process over outcome.
I try not to compliment someone’s shoes, I’ll instead say they have great taste.

I pride myself on giving good compliments, and a good compliment is more than just complimenting applied behavior.

A great compliment cannot be about anyone else and it cannot have been spoken by anyone else.

It’s non-fungible: an action that can’t be swapped, traded, or generalized. They are one-of-one.

II: The non-fungible reach-out

I was discussing networking with other professionals my age. Someone asked the question:

“How do you choose who to respond to when people reach out for a coffee chat?”

Someone else spoke up:

“I just want to see a little bit of effort, just personalizing the invite a bit to show we have something in common.”

I agreed. Similar to compliments, a good reach-out message cannot be about anyone else and it cannot have been sent by anyone else.

Here we find that non-fungibility again, let’s call this the non-fungible principle.

One of the best messages I’ve received was after a brief conversation at a networking brunch, they sent:

“It was great meeting you at brunch! I've since read nearly all of your essays and have been really enjoying them. If you're willing, I'd lov

...