trentmkelly/LessWrong-43k huggingface dataset:
Retrospective: 12 [sic] Months Since MIRI
by james.lucassen — 2025-01-21T02:52:06.271Z

it's now been 15 months since MIRI but I just remembered that three separate people have told me they liked this post despite my not cross-posting it, so I am now cross-posting it.

Not written with the intention of being useful to any particular audience, just collecting my thoughts on this past year's work.

September-December 2023: Orienting and Threat Modeling

Until September, I was contracting full time for a project at MIRI. When the MIRI project ended, I felt very confused about lots of things in AI safety. I didn't know what sort of research would be useful for making AI safe, and I also didn't really know what my cruxes were for resolving that confusion. When people asked what I was thinking about, I usually responded with some variation of the following:

MIRI traditionally says that even though their research is hard and terrible, everyone else's research is not at all on track to have a shot at making safe AI, and so MIRI research is still the best option available.

After working with MIRI for a year, I certainly agree that their research is hard and terrible. I have yet to be convinced that everyone else's research is worse. I'm working on figuring that out.

So I set out to investigate why MIRI thinks their research is the only kind of thing that has a shot, and if they're right or wrong about that. My plan? Read a bunch of stuff, think about it, write down my thoughts, talk to lots of people about it, try to get a foothold on the vague mass of confusion in my head. To my credit, I did anticipate that this could eat up infinite time and effort if I let it, so I decided to call a hard "stop and re-evaluate" at the end of September. To my discredit, when the end-of-month deadline came I took a moment to reflect on what I'd learned, and I realized I still didn't know what to do next. I thought that all I needed was a little more thinking bro and all the pieces would snap into place. Just a little more thinking bro. No need to set a next stop-and-re-evaluate deadline, it'll just be a little more.

This dragged out into two more months of ruminating, mapping out cruxes, trying out different operationalizations, distilling what I'd learned, doing dialogues with myself, all while my total productivity slowly ground to a halt. I didn't want to talk to people anymore, because I was upset and embarrassed about how little progress I had made. I started to dread sitti...
Easily Evaluate SAE-Steered Models with EleutherAI Evaluation Harness
by Matthew Khoriaty — 2025-01-21T02:02:35.177Z

EDIT: The folks at EleutherAI have taken and improved my work, and extended it to work with the Sparsify framework in addition to my use of SAE-Lens! I'm super proud that I contributed to that :)

That's me!

I haven't used the new system, but if you want to use it then the how-to is in EleutherAI's README.
Everything below here is outdated, only relevant for historical reasons / my portfolio of cool things I've done.

(Status: this feature is in beta. Use at your own risk. I am not affiliated with EleutherAI.)

Sparse Autoencoders are a popular technique for understanding what is going on inside large language models. Recently, researchers have started using them to steer model outputs by going directly into the "brains" of the models and editing their "thoughts" — called "features".

EleutherAI's Evaluation Harness is an awesome project which implements a wide variety of model evaluations including MMLU, WMDP, and others. You can now test SAE-Steered Models on many of those sets! (Not all of the evaluations are available right now for SAE-Steered models: just ones that only need 'loglikelihood'.) That includes multiple-choice questionnaires such as MMLU and WMDP.

As part of my research project in Northwestern's Computer Science Research Track[1], my team and I wanted to be able to easily evaluate steered models on benchmarks, MMLU and WMDP in particular. It was thanks to the AI Safety Fundamentals Course that I realized that I should put in extra effort to make the tool available to others and communicate how to use it. This constitutes my AI alignment project for the course.

How to Evaluate SAE-Steered Models

I've made a pull request to EleutherAI, so hopefully my contributions will be available there in the near future. If my work doesn't meet their criteria and they decline my code, then the steered model evaluation code will still be available on my fork here.

Step one:

If EleutherAI has approved my pull request, run these lines (from the installation section of the README).

git clone --depth 1 https://github.com/EleutherAI/lm-evaluation-harness
cd lm-evaluation-harness
pip install -e .

If not, run these lines

git clone --depth 1 https://github.com/AMindToThink/lm-evaluation-harness.git
cd lm-evaluation-harness
pip install -e .

Step Two: Deciding on your interventions

You first need to ask:[2]

Which model should I edit?
Which SAE-version should I use?
W

...
Why We Need More Shovel-Ready AI Notkilleveryoneism Megaproject Proposals
by Peter Berggren — 2025-01-20T22:38:26.593Z

A lot of people within AI safety and adjacent ecosystems have discussed AI notkilleveryoneism megaproject ideas, but there appears to be a key problem with these previous discussions. These discussions presuppose that the funding landscape will not change much in the near future, in ways that have proven inaccurate in the past (e.g. assuming that FTX funding would last forever). As such, their current assumption that there will not be enough funding for true megaprojects in the AI notkilleveryoneism space puts a damper on serious discussion of megaprojects. However, there is tremendous uncertainty surrounding funding and public awareness in the AI notkilleveryoneism space, even in the very near future. For this reason, planning for a wide range of funding environments would be prudent. This includes serious planning for a massive increase in funding.

If one thinks there is likely to be a massive increase in funding in the near future, one should focus a large amount of effort on writing shovel-ready AI notkilleveryoneism megaproject proposals. These proposals could not only serve to organize research in a high-funding world, but to serve as specific requests within the AI policy and governance ecosystem, and make a high-funding world more likely to come to pass.

Note: throughout this post I will use "AI notkilleveryoneism" to refer to research aimed at preventing AI from killing everyone, and "AI safety" more broadly to refer to any research aimed at making AI safer in the broadest sense of the word.

Reasons to anticipate a massive funding influx

Too often I see people conflating normative arguments on funding ("massive additional funding wouldn't do much good in AI notkilleveryoneism") with descriptive arguments ("AI notkilleveryoneism will not be receiving massive additional funding"). This seems deeply myopic to me, since it assumes that most AI notkilleveryoneism funding will always be coming from the Effective Altruism funding ecosystem.

The vast majority of money in the world is not controlled by the EA funding ecosystem. This is still true even if we're only talking about money that could in principle be allocated to AI notkilleveryoneism. For that reason, the EA community's view on funding constraints within AI is unlikely to be the primary ideological motivator for more or less funding in the field.

To that end, it is distinctly possible for t...
Tips and Code for Empirical Research Workflows
by John Hughes — 2025-01-20T22:31:51.498Z

Our research is centered on empirical research with LLMs. If you are conducting similar research, these tips and tools may help streamline your workflow and increase experiment velocity. We are also releasing two repositories to promote sharing more tooling within the AI safety community.

John Hughes is an independent alignment researcher working with Ethan Perez and was a MATS mentee in the Summer of 2023. In Ethan's previous writeup on research tips, he explains the criteria that strong collaborators often have, and he puts 70% weight on "getting ideas to work quickly." Part of being able to do this is knowing what tools there are at your disposal.

This post, written primarily by John, shares the tools and principles we both use to increase our experimental velocity. Many readers will already know much of this, but we wanted to be comprehensive, so it is a good resource for new researchers (e.g., those starting MATS). If you are a well-versed experimentalist, we recommend checking out the tools in Part 2—you might find some new ones to add to your toolkit. We're also excited to learn from the community, so please feel free to share what works for you in the comments!

Quick Summary

Part 1: Workflow Tips. We explain how to get the most out of your terminal with frameworks such as "Oh My ZSH" and how to easily deploy this on new machines with dotfiles. Learn about useful features in VSCode and what pre-commit hooks are.
Part 2: Useful Tools. Learn about software such as Tuple (for pair programming), LLM-assisted coding tools such as Cursor, and command line packages such as uv for Python package management.
Part 3: Experiment Tips. We explain tips for two research modes: "De-risk sprint mode" and "Extended project mode."
Part 4: Shared AI Safety Tooling Repositories. We release two repositories used by Ethan's MATS scholars: one for shared tooling and the other for examples of using the shared tooling.

Part 1: Workflow Tips

Terminal

Efficient terminal navigation is essential for productivity, especially when working on tasks like running API inference jobs or GPU fine-tuning on remote machines. Managing directories, editing files, or handling your Git repository can feel tedious when relying solely on bash commands in a standard terminal. Here are some ways to make working in the terminal more intuitive and efficient.

Terminals

Mac: We recommend iTerm2 .

...
Lecture Series on Tiling Agents #2
by abramdemski — 2025-01-20T21:02:25.479Z

It looks like my attempt to invite people to the lecture series failed, ie, the link I provided last time only invited people to the first event in the series.

Here is a new link for this week: https://calendar.app.google/U6yuY56feipJNvgp8

Here is the link for the video call, which should stay the same every week, but doesn't add the event to your calendar: https://meet.google.com/bwp-nkck-ros

I don't want to keep posting like this each week, so I'm interested if anyone knows how I can provide an invite link that invites people to the whole series (or something equivalently helpful for people).

This week I'll keep going through the slides I started last time. Here is the video of last week's session: https://drive.google.com/file/d/1UU3paAmJ9J4FdlJ6LIz9gvm7wcw53W06/view?usp=sharing
Announcement: Learning Theory Online Course
by Yegreg — 2025-01-20T19:55:57.598Z

The application deadline for the course has now passed. We received a very promising number of submissions! Feel free to continue discussion in the comments below.

Hey everyone! Gergely and Kōshin here from ALTER and Monastic Academy, respectively. We are excited to announce a 6-week, 4-hours-per-week mathematics course designed to introduce a few early topics in what we are calling mathematical AI alignment (roughly, classical learning theory up to embedded agency and infra-Bayesianism, though this course will cover just an initial portion of this) aimed at folks with the math proficiency level of a university math major (formal university degree not required). What this means is that if you have some mathematical dexterity, and wish that you could answer difficult questions on MDPs, POMDPs, learnability, sample complexity, bandits, VC dimension, and PAC learning, then this course may be for you.

Lectures will be given over video call by Gergely, who has a math PhD from Stanford and has been working with Vanessa Kosoy over the past two years on the Learning Theoretic Agenda. Kōshin is helping with logistics. We are planning to have a one-hour video lecture each week, a one-hour discussion session each week, and a weekly problem set that should take around two hours to complete.

The cost of the course is $200 per participant, which we’ll refund up to the full amount based on attendance.

There is a question set for which we ask you to submit solutions in order to apply to join the course. This is to help us select a cohort with reasonably uniform mathematical proficiency, and to help us design material for that level of proficiency. We are looking for a cohort who are willing to make a strong commitment to joining all the lectures and discussion sessions, and solving all the question sets for the entire 6-week duration. You do not need to make that commitment in order to apply, but we’ll ask for it before the course begins.

To get started right away, see the entry questions and apply for the course.

The remainder of this post is organized as follows: First, we’ll pitch the larger vision for the course. Second, we’ll discuss the entry question set and why it makes sense for anyone interested in the course to submit their best attempt at it. Third, we’ll go over dates and costs.

Pitch for the course

We feel that the best way to pitch the course is to present the ques...
The Hidden Status Game in Hospital Slacking
by EpistemicExplorer — 2025-01-20T18:35:54.086Z

Why do highly-paid hospital workers slack off and complain so often? Most would say "because they can" or "they're just lazy" or "it's a tough job, stress release." But I suspect there's a deeper status game at play - one that may illuminate broader patterns of institutional decay.

Consider: I recently observed an ICU Registered Nurse who makes at least ~$236K/year, in a highly rated hospital in San Francisco, demonstrating conspicuous low effort. They:

Openly bragged to 6 people in the 1 hour of my observation about how minimal of work he was doing
Actively discouraged others from working hard
Optimizing for maximum visibility while achieving peak gravitational slouching ability
Consumed unhealthy food conspicuously.

The conventional view says this is just poor work ethic or a "burned out" employee. But notice - they're not just working little, they're signaling how little they work. This is key.

In most workplaces, appearing hardworking is high status. But here we see the opposite - there's status in showing how little you care. Why?

I suspect it's about demonstrating market power. By conspicuously slacking while keeping their high-paying job, they signal: "I'm so valuable that I can get away with this." It's like a peacock's tail - wasteful display that proves fitness.

The unhealthy food in the hospital setting amplifies this signal: "I'm so secure in my position that I don't even need to maintain appearances."

But there's more. By discouraging others from working hard, they're engaging in a subtle status competition. Each person working hard makes their slack more conspicuous. By convincing others to slack, they maintain their relative status position.

This matches broader patterns. High-status people often signal by breaking rules that bind others. Think of celebrities wearing ripped clothes or tech CEOs in hoodies.

The Status-Wage Paradox

If this model is right, simply raising wages won't help - it might even make it worse by increasing the status value of conspicuous slacking. The more you're paid, the more impressive it is to visibly slack off.

Connection to Known Failure Modes

This pattern connects to several institutional failure modes:

Moloch Dynamics: The situation resembles a multi-polar trap where individual incentives (status through slacking) create collectively harmful outcomes (degraded care quality).
Principal-Agent Problems: Classic monitoring issue

...
Monthly Roundup #26: January 2025
by Zvi — 2025-01-20T15:30:08.680Z

Some points of order before we begin the monthly:

It’s inauguration day, so perhaps hilarity is about to ensue. I will do my best to ignore most forms of such hilarity, as per usual. We shall see.

My intention is to move to a 5-posts-per-week schedule, with more shorter posts in the 2k-5k word range that highlight particular subtopic areas or particular events that would have gone into broader roundups.

This means that the Monthly Roundups will likely be shorter.

If you’re considering reading Agnes Callard’s new book, Open Socrates, I am reading it now and can report it is likely to get the On the Edge treatment and its own week, but of course it is too soon to know.

I may be doing some streams of myself working, via Twitch, primarily so that a volunteer can look for ways to build me useful tools or inform me of ways to improve my workflow. You are also of course welcome to watch, either live or the recordings, to see how the process works, but I make zero promises of any form of interaction with the audience here. I also might stream Slay the Spire 2 when the time comes, once I have access and they permit this.

On with the show.

Table of Contents

Bad News.

Wanna Bet.

A Matter of Trust.

Against Against Nuance.

Government Working.

Scott Alexander on Priesthoods.

NYC Congestion Pricing Bonus Coverage.

Positive Sum Thinking.

Antisocial Media.

The Price of Freedom.

Mood Music.

Dedebanking.

Good News, Everyone.

While I Cannot Condone This.

Clear Signal.

When People Tell You Who They Are Believe Them.

What Doesn’t Go Without Saying.

Party at My Place.

I Was Promised Flying Self-Driving Cars.

Gamers Gonna Game Game Game Game Game.

For Your Entertainment.

Sports Go Sports.

The Lighter Side.

Bad News

PornHub cuts off Florida, the 13th state to lose access, after Florida passed an age verification law, and PornHub quite reasonably does not want to ask for your ID.

Running for Congress is a horrible deal and essentially no sane person would do it. If we want good people to run for Congress and for them not to consider sleeping on cots in their office we need to dramatically raise pay, which we should do.

Share of American adults having dinner or drinks with friends on any given night has declined by more than 30% in the past 20 years, and changes from Covid mostly have not reversed.

Older software engineers often report that once they...
Things I have been using LLMs for
by Kaj_Sotala — 2025-01-20T14:20:02.600Z

There are quite a few different things you can use LLMs for, and I think we’re still only discovering most of them. Here are a few of the ones I’ve come up with.

My favorite chatbot is Claude Sonnet. It does have a tendency for sycophancy – for example, it will go “what a fascinating/insightful/excellent/etc. question!” in response to most of the things you might ask it. Some people find this annoying, while my brain just filters it out automatically. If you don’t like it, you can put in a custom instruction telling it to do something else.

Also, a tip from Alyssa Vance: “when talking to Claude, say that your idea/essay/code/etc. is from your friend Bob, not you. That way it won’t try to blindly flatter you”.

Uses

Creativity

Essay brainstorming. I’ll tell Claude “here’s an essay that I started writing” and copy-paste what I’ve written so far to it. It will comment with ideas, possible other directions, and connections to related things.

Then I have a conversation with it and also tell it about other ideas I want to work into the essay, but haven’t written yet. Sometimes I’ll ask it things like “here’s an idea I’d like to express but this phrasing feels clunky, would you have better suggestions”.

In the end, I copy large chunks of the conversation (both things that I explained to it, and ideas that it had in response) directly into a text document and edit them into a smooth essay.

Role-playing/fiction-writing. I like to start by writing a brief biography of a character I’m drawn to, and then telling Claude something like “please analyze this character’s psychology and note anything about them or the setting that the description implies but doesn’t state outright”.

Then we brainstorm things about the characters and the setting that seem interesting, and at some point we transition to writing prose, with me writing some of the characters and Claude writing the others.

Emotions and introspection

Introspection aid. Often when I have some unpleasant feeling I don’t quite understand or I’d like to try shifting, I tell Claude “I’ve got a feeling of [sadness/anxiety/loneliness/whatever] I’d like to explore, I’d like you to guide me in using Gendlin’s Focusing (and its variants) to explore it”. Excellent Focusing facilitator and good at noticing patterns and asking questions.

Lifting mood. One evening, I was in a bit of a depressive f...
What are the chances that Superhuman Agents are already being tested on the internet?
by artemium — 2025-01-20T11:09:33.835Z

I keep seeing odd bits of information from the grapevine about how Superintelligent Agents are on the horizon, with the latest being this Axios article. While I’m still unsure what to think, I started considering the possibility that these agents might already exist in an early form and are being tested online in secret. If that’s the case, could we detect them somehow?

Here are a few possible domains where such agents might already be tested:

1. Anonymous High-Performing Investors
Through platforms that allow pseudonymous accounts to trade stocks, cryptocurrencies, or other assets, an AI agent could potentially execute trades with an unusual level of insight and timing. If one were to notice an otherwise account consistently achieving market-beating returns—beyond what can be explained by luck or insider knowledge—it might hint at something like this.

2. Dominant New Gaming Accounts
We have already seen AI research before, such as OpenAI’s Dota AI or DeepMind’s AlphaStar Starcraft AI. A superhuman AI tested covertly on the internet might show up as a new account with inexplicable performance. For example, a sudden top-ranking player with flawless tactics. Although cheats and bots already exist, a genuine superhuman AI might adapt in real time, learn strategies from human opponents, and systematically dominate high-level competition without obvious cheating.

3. Mysterious GitHub Repositories with Pristine Code
Imagine a newly created GitHub repository featuring code that is exceptionally clean, well-structured, and updated at a rapid pace—perhaps multiple substantial commits in a single day, all made by a single, seemingly tireless account. Moreover, these updates might involve sophisticated algorithms or solutions to problems that typically require large teams or specialized domain expertise.

4. Other Platforms
Competitive programming websites, hacking contests, puzzle-solving communities, e-sports tournaments, or even social media accounts that consistently produce highly engaging content without a predictable "bot" posting style could, also be testing grounds.

There are probably some methods one could use to analyse those domains for outlier performance, but I’m not entirely certain the premise holds water. Thoughts?