trentmkelly/LessWrong-43k huggingface dataset:
Interpretable Fine Tuning Research Update and Working Prototype
by Matthew Khoriaty — 2025-05-16T03:44:03.706Z

Status: Very early days of the research. No major proof of concept yet. The aim of this research update is to solicit feedback and encourage others to experiment with my code.

GitHub repository

Main Idea:

Researchers are using SAE latents to steer model behaviors, yet human-designed selection algorithms are unlikely to reach any sort of optimum for steering tasks such as SAE-based unlearning or helpfulness steering. Inspired by the

Bitter Lesson

, I have decided to research gradient-based optimization of steering vectors. It should be possible to add trained components into SAEs that act on the latents. These trained components could learn optimal values and algorithms, and if we chose their structure carefully, they can retain the interpretable properties of the SAE latent itself. I call these fine-tuning methods Interpretable Sparse Autoencoder Representation Fine Tuning or “ISaeRFT”.

A diagram of an interpretable fine tuning component acting on the interpretable latents of a sparse autoencoder. The drawing depicts a feed forward neural network, though my recent research has focused on learning scaling factors to apply to features.

Interpretable components that act on SAE latents contribute to a number of safety-relevant goals:

Data Auditing

By seeing which features change during training, it should be possible to check that there aren’t harmful biases in the data or data poisoning. I hope this method can help pin down the cause of

Emergent Misalignment

and identify when it occurs.

Validating SAE interpretation methods

If current SAE interpretation methods do a good job of reflecting their actual meaning to the model, then that should be reflected in which latents are changed during learning.
For example, if you train a model on helpfulness text using ISaeRFT, but when you look at the trained components, features with labels that make them seem irrelevant are changed and features you would expect to be changed are unchanged, then this indicates a problem with the labeling method or the causal relevance of those features.

Understanding safety-relevant behavior

ISaeRFT ties nicely to reinforcement-learning based methods for guiding model behaviors, allowing us to understand what these methods taught the model. It can also be used to see what a dataset teaches a model.

Model diffing
If ISaeRFT fails to have meaningful successes on downstream tasks

...
It Is Untenable That Near-Future AI Scenario Models Like “AI 2027” Don't Include Open Source AI
by Andrew Dickson — 2025-05-16T02:20:48.115Z

Note: This post is the second in a broader series of posts about the difficult tradeoffs inherent in public access to powerful open source models (the first post is

here

). While this post highlights some dangers of open models and discusses the possibility of global regulation, I am not, in general, against open source AI, or supportive of regulation of open source AI today. On the contrary, I believe open source software is, in general, one of humanity’s most important and valuable public goods. My goal in this series of posts is to call attention to the risks and challenges around open models now, so we can use the time we still have before risks become extreme, to collectively explore viable alternatives to regulation, if indeed such alternatives exist.

In the past year there have been a number of impressive efforts by AI safety researchers to create detailed “scenario models” that represent plausible stories for how the next several years might play out. The basic idea of these models is that while they won’t get everything right, the scenarios should at least be plausible in the sense that they are internally consistent and are at least mostly believable in terms of how the events unfold in the story. The best and most popular and best-developed examples of such scenario models that I’m aware of are Leopold Ascherbrenner’s

Situational Awareness

, @joshc’s

How AI Might Take Over in 2 Years

and recently

AI 2027

by @Daniel Kokotajlo et al.

I have personally been very much inspired by all three of these efforts and believe they’ve had some of the greatest impact of any AI safety effort so far, especially in terms of raising public awareness around AI risks. However there is an aspect of such scenario models that I believe is – for lack of a better word – untenable, which is that so far they have either entirely ignored, or vastly downplayed the role open models are likely to play as the near future of AI unfolds.

Broadly-speaking, the three scenarios above focus almost entirely on either the AI arms race-style dynamics between leading nation-states (particularly the US and China) or the efforts of one or more leading labs to internally align their most powerful models, or both. The result of this is that the scenarios tend to portray essentially all key high-level global impacts from AI over the next several years as c...
Apply to Visiting Fellows at Constellation, due June 13
by Ella Markianos — 2025-05-16T02:20:14.929Z

Applications just opened for Constellation’s

Visiting Fellows

program.

Constellation is a research center in Berkeley. We host people who are working to help the world safely navigate the development of transformative AI. Our office houses great orgs and researchers in several important areas of AI, including technical safety, governance, and forecasting.

Visiting Fellows is a 3-6 month program for people working on Constellation’s

focus areas

who haven’t spent much time in our office before. It’s a chance to work out of Constellation with travel, housing, meals, and office space covered. The program is intended to make it easier for people outside of the Bay Area to visit and engage with Constellation’s community.

Why should I do it?

Working at Constellation is one of the fastest ways to get real context on the field of AI safety–and to accelerate the work you’re already doing. If you don’t normally work or live in the Bay Area, visiting Berkeley is a great way to encounter leading research, ideas, and people in AI safety.

Constellation makes it easy to have meaningful conversations and collaborate on important work. Some of the most valuable discussion happens informally over lunch or dinner, which is served in-office every day. It’s not unusual for someone to share what they’re working on over lunch, and for someone else to chime in with a great idea or offer to help—and before long, they’re working on something together. There’s also something great about being able to poke your head into someone’s office and get their take in real time.

Constellation is a great place to work: the office space is beautiful and conducive to productivity, and free lunch and dinner optimized for nutrition and deliciousness is served daily!

More about what Constellation is:

Constellation is an independent research center that brings together people throughout the AI safety ecosystem to accelerate insight, research, and talent through better cooperation. Unlike a conference or summit, Constellation operates continuously, in the form of a physical workspace as well as conference-style talks, workshops, and training bootcamps.

Our shared workspace hosts over 100 people per week across dozens of AI safety organizations in nonprofits, academia, industry, and government. Hundreds of other researchers spend time at Constellation for shorter visits eac...
Staying in a Capsule Hotel
by jefftk — 2025-05-16T00:20:01.205Z

A hotel room generally feels way bigger than I need when traveling:
I'm really just looking for somewhere to sleep. When I first read the

Wikipedia page
for capsule hotels, maybe 20y ago, this seemed like a very
reasonable approach: lots of people could sleep in a small space,
without the downsides of open hostels. I recently travelled to DC for
work and found the

U Street
Capsule Hotel. Seemed worth trying!

For $82 I got a capsule for a Thursday night:

Some capsules were perpendicular to the hall, which seems a bit more
awkward for getting in and out but would use the space more
efficiently:

They were really very blue. Here's a photo from the inside:

Seems like they were designed by people who didn't know or didn't care
that blue light is bad for sleep? You could turn off the light,
though, so it wasn't that bad.

The thing that I
wasn't expecting, however, was the minimal sound isolation. Before
trying one of these I had assumed that the point of a capsule instead
of bunk beds was noise isolation. While the other people in the room
did a good job of being quiet, thin walls plus a window shade for a
door did very little to keep out noise. Something more like a
horizontal call booth (with active
ventilation, and perhaps a failsafe to automatically open on power
loss) would be a lot better.

The ventilation was also hard to control: it was somehow connected to
the lights, where I accidentally turned the air off when turning the
lights off and didn't realize until ~1am when it got pretty stuffy.
This would probably get better with practice, but it could be a lot
more intuitive.

Overall it was ok and I might do it again, especially if it were a lot
cheaper than other options, but I liked it significantly less than I
expected to.

Comment via: facebook, mastodon, bluesky
Researching Synthetic Consciousness: sound appealing? 
by Brad Dunn — 2025-05-15T22:29:05.230Z

Without going into the philosophical or technical underpinnings too much, several individuals I know from a wide range of backgrounds—computational neuroscience, computer science, philosophy, AI research—have put together a project where we hope to put some real energy towards studying consciousness in synthetic environments.

Obviously, if this is an area of interest, you'll likely have questions about whats possible given the landscape of consciousness studies, let alone extending that into non-human systems, but right now its nothing more than a group of individuals, being wrapped up into a not-for-profit research body who are;

passionate about the topic
keen to put some real energy towards it.
Have some time to give.
And hopefully push things along.

I've set up a very simple website just to collect some interest, but given the community here I thought i'd see if people were interested in being involved. As of right now, we have about 10 people committed to the project in various capacities.

We have some positions available that are a bit more clearly defined, are currently talking to some individuals who will make up the board of directors, and are hoping to run a kind of research fellowship program where we can put some real meat on the bones. The intake for fellowships will begin in about 8 weeks time (July, 2025).

If this topic, or doing research in this space, sounds interesting. I'd love to hear from you.

More details at https://alternativemind.org
Starting Over: What to tell Sarah, at the edge of professional oblivion. 
by Brad Dunn — 2025-05-15T21:34:41.224Z

I came across a sad post recently from a woman on LinkedIn— let’s call her Sarah—who spent over a decade, in her words, as a top 2% recruiter, and was now finding it impossible to land even the most basic of roles. It was one of those earnest, help-me / trauma-essays, with the comments below full of support, shock that Sarah was still unemployed, and love-heart emojis — the most potent form of emotional currency we offer, always in silence.

I feel for Sarah, who is part of a growing group of professional refugees flooding our feeds, as the snow globe of the global economy is shaken; tariffs, AI, and layoffs — shake, shake, shake. When I imagine myself in Sarah’s shoes — what it would be like, in that moment, as she drafts the words — it feels like a mixture of humility, shame, and survival-driven desperation, the latter of which makes the hairs on my arms stand tall. This late-stage survival instinct perfectly demonstrates how tough human beings can be, but watching it unfold on social media makes me very sad, for we are so very fragile, so easily put on tilt. Perhaps these posts are ‘a reminder to consider yourself lucky’, I tell myself, or maybe some kind of omen. Either way, there are more of them, and more every day.

Professional MADAY

Reading these posts has me conflicted. Instinctively, the system 1 part of my brain feels for Sarah and her kin. Yet it’s also confronting to admit there’s an element of judgment too. The more calculating part of me, floating somewhere between my prefrontal cortex and my amygdala, tries to model what led them here — what miscalculation occurred — and perhaps, what can be done about it now; solution mode, as they say. We all do this, on some level. We recognise the plight of others and overlay it on our own circumstances. Not only do we notice the misery in the world, but knowing that out there, it’s bad, tells us something about how we are doing by comparision. Misery is a feedback loop, of sorts. I did not know Sarah, so I of course did nothing, the algorithm was simply ‘letting me know’ I might be interested in this car accident. Once I’ve read her story, in a bleak way, I have internalised her plight and suffering for my own purposes, for content, to spew back out, which is how you ended up here. You also, in a way, have Sarah to thank. This loop of misery, pity and judgement, has a darkness about it. Yet ultimately, I say...
Tax-Optimized Risk in Portfolio Allocation
by Brendan Long — 2025-05-15T18:53:13.959Z
Link: https://www.brendanlong.com/tax-optimized-risk-in-portfolio-allocation.html

When you're subject to capital gains taxation, the government shares in some of the upside, but when you have capital losses, the government shares in the downside too[1]. Because of this, the actual risk (and reward) of any given portfolio is lower than it seems. To counteract this, you should consider shifting your allocation toward riskier assets.

This was new to me, but you can find articles on this subject going all the way back to "Proportional Income Taxation and Risk-Taking" by Evsey D. Domar and Richard A. Musgrave in 1944.

The Mathematical Framework

Take a tax rate, like the 20% long term capital gains rate in the United States. For any investment subject to this tax, the gains are 20% lower than they would be without the tax, but the losses are 20% lower as well (so 80% of what you would naively expect). To increase that back to 100% of the desired risk/return, we just need to increase both risk and reward by 25% (1 / 0.8 = 1.25).

For short-term gains taxed at ordinary income rates (potentially 37%+), the effect is even stronger. The optimal risk increase would be approximately 59% (1/(1-0.37) = 1.59).

Practical Implications

This is unfortunately complicated. The straightforward takeaway is that if your 60/40 stock/bond portfolio is subject to the 20% long-term capital gains tax, you should increase the stock by 25% and shift to 75/25. This isn't exactly right though, since the two assets differ in more ways than their risk profile. My best guess is that stock's tax-advantaged treatment in other ways make the 60->75 optimal shift an underestimate[2], but maybe don't change how you invest your life savings without more research.

Conclusion

The tax code's treatment of capital gains and losses means that it's rational for investors to take increased risk than they would without taxes, since the government will share in both the gains and the losses.

^

If you have capital losses, you can use those to offset capital gains, which reduces their "badness" by the capital gains tax rate. For example, if you have $100 of capital losses, at some point you will be able to apply those to cancel $100 of gains and avoid $20 in taxes, so in some sense you've only really lost $80. But yes, this oversimplies somewhat.

^

Stocks are subject to the 20% long-term capital gains tax, but bond income is subject to ordinary income tax. The latter is much higher, so even without the

...
Some skepticism about skepticism about efficacy of pausing AI
by extinction-bounties — 2025-05-15T18:15:41.298Z

TL;DR: Basic economics to AI research suggests that making frontier R&D more expensive will reduce its pace, shifting our median timelines outward and lowering FOOM risk. This is probably true even if only one major geopolitical player does it.

It is a truth universally acknowledged in Econ 101 classes that an action made more expensive by some means (costlier inputs, government regulations, etc.) will be done by people less often. Yet, for some reason, people often forget they can apply this to the action of producing novel scientific or technological research - something I claim is probably still the dominant FOOM risk factor of today.

(I don't plan to back up that claim rigorously here today. For now I will simply (a) point out that figuring out good boundaries to any boundary problem is often harder than solving the problem itself, and (b) clarify that I do consider training new more advanced frontier models to be a pretty good starting point for what counts as 'novel scientific or technological research' in my definition, even though you could well argue that by this point it's just an implementation detail. If you think, however, that e.g. we're a few API wrappers or Selenium sessions away from turning GPT-o3-pro into an actual FOOM risk, well, it's certainly possible, but I find myself doubtful of that.)

Research has to be done by someone[1]. There's no a priori reason to suspect that that someone is granted protection from that Econ 101 principle just because we rightfully consider research to be a vaunted profession. History overwhelmingly supports the hypothesis that when research is very expensive, relative to other things one could do with their time, it simply doesn't get done. At the extreme end, Homo sapiens with cognitive capabilities similar to where we are today first hit the evolutionary scene about 300,000 years ago, but the majority of scientific research actively discussed today is from the last ~0.1% of that time we have spent post-Industrial Revolution.

All of this adds up to a very boring hypothesis: If the act of doing AI research become more expensive for people, then less of it will be done, and our chances of entering FOOM will be lower, or equivalently, our median AI timelines should shift outward. Boring is good! We like boring hypotheses around here.

I often hear criticism and skepticism of the "AI policy" and "Pause AI" folks,...
time is event based
by thiccythot — 2025-05-15T18:07:43.811Z

After college I worked for a high frequency trading firm where the basic unit of time was the microsecond. A single blink—about 250,000 µs—was long enough for our algorithms to open and close thousands of positions, each trade born and buried long before a human observer would have even noticed the screen had changed.

At the end of each day we would grade our trading by looking forward into the future for every trade. If we bought stock A at $5.00, we would have liked to see the market trading at $5.01 moments later.

We didn’t use many time-based benchmarks. 10 seconds for a stock like NVDA was an eternity compared to a no-name stock. Thousands of individual trades could have been entered and exited in NVDA in the time it took the no-name stock to update it’s price a single time. There were levels to this eye blinking metaphor.

We used event benchmarks instead of time benchmarks. Every time someone executed a trade, cancelled a bid, updated an offer, was counted an event. Any change in the state of the order book reflected someone pricing in new information about the world. We would then benchmark our trading to the market price say 1000 events in the future. For some stocks, this would be milliseconds. For others, minutes. This normalized our performance benchmarks across all different types of stocks and made us more effective in generalizing our strategies.

It was my first lesson that event based time perception mattered more than subjective time.

This concept carried with me as I left HFT and moved into crypto discretionary trading. Crypto is the only market that trades 24/7, meaning there simply was no rest for the wicked. The game was less about brilliance and more about being awake when it counted. Resource management around attention and waking hours was a big part of the game. I quickly learned there were only a few important hours that I needed to be awake.

Pretty quickly I mapped the day’s “hot zones”:

7 – 11 am ET US open

3 – 4 pm US Close

4 – 5 pm After-hours headline dump

7 – 8 pm Quants rebalance at 0 UTC

9 – 11 pm Korean retail pumps altcoins

4 – 7 am Euros position for US open

Everything outside those windows was mostly noise. Weeks where nothing happened followed by a single headline turned into minutes where weeks happened.

My cofounder and I developed a polyphasic sleeping routine so that we would be conscious during as many of the...
Consider Others' Cost Tolerances
by nomagicpill — 2025-05-15T17:43:24.427Z
Link: https://nomagicpill.github.io/thoughts/costtolerance.html

Just because you won't, doesn't mean they won't.

Background

There are a few main categories of costs associated with everyday decision-making:

Time
Money
Physical suffering
Mental suffering
Risk of death

Everyone has their own limits for each that must be calculated for every situation that pulls from that category's coffers. (I view this is different from an opportunity cost.) Only if it is below the threshold for each can the action be done. For example, I may be willing to put up enough money and time for the glory of summitting K2, but my death risk tolerance is much lower than its ghastly 13-23% fatality rate, so I say no.

This is straightforward and easy to do for a single individual, but becomes much more opaque when more people are involved, especially when only one person has all the information.

Examples

I: Time

John goes on an African safari with Clarence on a whim, doing little research and spending $10,000 and 10 days of his precious PTO. He trusts it will be awesome based on Clarence's previous trip reports. He is excited for constant action with animals and the local culture. He quickly realizes that seeing the "cool" animals—lions, leopards, and such—is not guaranteed and that most days are coupled together doing the same thing in case something isn't seen the first time (e.g., days 2 and 3 are the same, 4 and 5 the same, etc). He would have spent at most $5,000 and 5 days of PTO, not $10,000 and 10 days.

John underestimated how much Clarence values seeing these animals, even if only for a short amount of time. In turn, he underestimated the cost Clarence was willing to incur to make this happen.

II: Money

Brett happily agrees to go out for a "nice dinner" with his friend Samuel despite not being very into food or drink. Samuel picks Brett up and they carpool to a "pretty good spot" where a valet whisks the car away while the two men head in. Brett is astounded at the prices, estimating he'll have to spend around $150 to not look stingy in front of his new friend—a nice dinner to him was at most $50! Meanwhile, Samuel is salivating over the menu and ripping off questions to the waiter about umami this and confit that.

Brett underestimated how much Samuel likes to fine dine and had to pay for it.

III: Physical Suffering

Amy accepts an invitation from her friend Sonia to help her prepare her front yard for a garden, something she only cares to do because she is ...