trentmkelly/LessWrong-43k huggingface dataset:
You don't have to be "into EA" to attend EAG(x) Conferences
by gergogaspar — 2025-03-20T10:44:12.271Z

I have recently shared a post about upcoming x/s-risk events, but I wanted to highlight this general part, as I often come across this (understandable) misconception about the scope of EAG(x) conferences when I talk to rationalists/AIS people.

Here is a meme I would like to spread:

People who are interested in AI Safety, but not EA, often get the impression that EAG(x) conferences are not for them.

To make it clear, you don’t have to be interested in EA to attend, and you should make this clear to other Lesswrong readers as well. Much of the content will focus AI Safety, and many people (including myself) are only attending to network with others in AIS. For better or worse, currently, the best place where you can regularly network with AIS people is at EA conferences. You can read the target about the target audiences for these conferences here, just replace the terms such as "working in EA" with "working in AI Safety".

Having said all that, if you actively dislike EA and EA culture, these conferences are probably not for you.

If you think there should be more AIS-specific conferences, I agree, so let's make them happen.
Defense Against The Super-Worms
by viemccoy — 2025-03-20T07:24:56.975Z

I am not writing this to give anyone any ideas. However, it seems to me that hackers are going to be able to leverage distributed language models in the near future, and I think that we should be creating proactive countermeasures before the tech is powerful enough to cause real damage. I predict that with globally distributed language model training having been proven possible, we will soon see bot-net style distributed inference. If you have enough devices on a home network, it is already trivial to run a very large model with the compute distributed across your phone, iPad, laptop, etc.

In my opinion, it is only a matter of time before hackers leverage distributed model inference to create self-propagating Super-Worms. These Super-Worms would function similar to a normal bot-net, with the added functionality of being able to recurse over their own code when they notice chunks of their compute being successfully eradicated by new additions to antivirus software.

Because the compute would be distributed, it seems obvious that the virus would be able to run more than just a single 405b-parameter model across the entire network. Given enough time, it could run many models in parallel, synchronously (or asynchronously) communicating with one another to improve their ability to spread. The virus, whose purpose would likely be to mine bitcoin or collect user data, would then be able to cybernetically adjust its own priorities. Should it dedicate more compute to self-improvement? Or, if the anti-virus software developers seem to be lagging behind, should it should dedicate more compute to mining bitcoin?

Of course, hackers won't be the only ones who can launch these distributed Super-Worms into the ether. Nation states, collectives, and anyone with access to the internet will be able to take one of these distributed viruses and tailor its ethos to their own desires. This, to me, seems to be the biggest risk with improving AI technology. Not a singleton deciding its time to off the entire planet, but smaller intelligent viruses wreaking havoc on our infrastructure.

My proposal for a solution here is an unfortunate one. I think the time of the world wide web is coming to an end. Smaller intranets are going to be the only way to keep our infrastructure safe. There should not be a pathway of fiber optic cables between a power plant and a hackers hideout. In many cases, t...
Socially Graceful Degradation
by Screwtape — 2025-03-20T04:03:41.213Z

Prerequisites:

Graceful Degradation

. Summary of that: Some skills require the entire skill to be correctly used together, and do not degrade well. Other skills still work if you only remember pieces of it, and do degrade well.

Summary of this: The property of graceful degradation is especially relevant for skills which allow groups of people to coordinate with each other. Some things only work if everyone does it, other things work as long as at least one person does it.

I.

Examples:

American Sign Language is pretty great. It lets me talk while underwater or in a loud concert where I need earplugs. If nobody else knows ASL though, it's pretty useless for communication.
Calling out a play in football is pretty great. It lets the team maneuver the ball to where it needs to go. If nobody else on the team studied the plays though, then the quarterback can call all the numbers he wants but nothing useful is going to happen.
Encrypting my email is pretty great. It lets me send messages to other people without the message getting intercepted. If the person I'm talking to ever leaves an unencrypted copy lying around or forwards the plaintext though, all that caution fails.

The important detail of these examples I'm trying to point at is that it doesn't matter if you're amazing at the skill if other people aren't. No amount of individual practice will make ASL a fluid tool for being understood in a loud concert or in spite of being hard of hearing. It's fundamentally a stag hunt, where it's only worth doing if other people are going to do it to.

What are some counterexamples?

Learning to read and write is pretty great. I can write notes to myself and by doing so extend my memory. I can use writing an essay to refine my thoughts on a topic. If other people know how to read and write that's great since we can write to each other, but even if I'm the only literate person in the world literacy is still useful.
Good data backups are pretty great. I can get into a habit of using extra hard drives or off-site backups to make sure I don't lose as much of my writing or family vacation photos if something bad happens to my computer. If other people also take good backups, that's nice because it means they might have backups I didn't think to make (the internet archive is good!) but even if I'm the only person in the world who makes backups this is still usef

...
Apply to MATS 8.0!
by Ryan Kidd — 2025-03-20T02:17:58.018Z

Applications

are open for the

ML Alignment & Theory Scholars (MATS)

Summer 2025 Program, running Jun 16-Aug 22, 2025. First-stage applications are due Apr 18!

MATS is a twice-yearly, 10-week

AI safety

research fellowship program

operating in Berkeley, California, with an optional 6-12 month extension program for select participants. Scholars are supported with a research stipend, shared office space, seminar program, support staff, accommodation, travel reimbursement, and computing resources. Our

mentors

come from a variety of organizations, including

Anthropic

,

Google DeepMind

, OpenAI,

Redwood Research

, GovAI,

UK AI Security Institute

, RAND TASP,

UC Berkeley CHAI

,

Apollo Research

, AI Futures Project, and more! Our

alumni

have been hired by top AI safety teams (e.g., at

Anthropic

,

GDM

,

UK AISI

,

METR

,

Redwood

,

Apollo

), founded research groups (e.g.,

Apollo

,

Timaeus

,

CAIP

,

Leap Labs

), and maintain a dedicated support network for new researchers.

If you know anyone who you think would be interested in the program, please recommend that they

apply

!

Program details

MATS is an educational seminar and independent research program (generally 40 h/week) in Berkeley, CA that aims to provide talented scholars with talks, workshops, and research mentorship in the fields of

AI alignment

, security, and governance, and connect them with the San Francisco Bay Area AI alignment research community. MATS provides scholars with housing in Berkeley, CA, as well as travel support, a co-working space, and a community of peers. The main goal of MATS is to help scholars develop as AI alignment researchers. You can read more about our theory of change

here

.

Based on individual circumstances, we may be willing to alter the time commitment of the program and arrange for scholars to leave or start early. Please tell us your availability when applying. Our tentative timeline for the MATS Summer 2025 program is below.

Scholars will receive a USD 12k stipend from

AI Safety Support

for completing the Training and Research Phases.

Applications (now!)

Applications open: Mar 19

Applications are due: Apr 18

Note: Neel Nanda's applications follow a modified schedule and are now closed.

Research phase (Jun 16-Aug 22)

The core of MATS is a two-month Research Phase. During this Phase, each scholar spends at least one hour a week wor...
Improved visualizations of METR Time Horizons paper.
by LDJ — 2025-03-19T23:36:52.771Z

TLDR: Skip to the last image at the bottom of my post to see the visualization.
I don't usually post here, but I know you folks like forecasting and timelines, so I think you might appreciate this. The recent time horizons paper by METR is great, but it only visualized tasks that models could complete with 50-80% accuracy. I think it's highly useful, and arguably even more useful, to see when capabilities would hit 95% or 99% accuracy or beyond for various time horizons.
So I went through the paper with the goal of getting the information that will allow such a visualization to be created, and I found distributions shown of model-specific accuracy levels for specific time horizon buckets.
I noted these down and noticed there is similar, if not identical slopes for those higher accuracy levels compared to the lower ones when plotted against time horizons(they also noted in the paper how the slope for 80% accuracy was conveniently very similar to the slope for 50% accuracy too), So this makes the visualization math easier as well, I keep the slope steepness the same while just needing to apply the appropriate Y-axis adjustment relative to release date.
I noted down the multiple difference between the time horizons of 50% accuracy, 80% accuracy, 95% accuracy and 99% accuracy, and did this for the following 6 models:
- Claude-3.7-sonnet
- Claude-3.5-sonnet(new)
- O1
- GPT-4o
- Claude-3-Opus
- GPT-4-0314
The paper mentioned a 5X multiple difference they found between 50% and 80% accuracy, and when I averaged my numbers for these 6 models I arrived at the same figure for 50% to 80%, as for the other accuracy multiples, it looks like this:
Finally, here is the visualization I ended up making, with expanded x-axis, as well as the short-term trend slope added of reasoning models for each of the new 80%-99% accuracy trends, which was also mentioned in the original paper (doubling every 4 months with the data points of 2024-2025)
I think there is good reason to believe that the short term trend of 4 month doublings may be over optimistic, but I can also see many arguments as to why the long term 7 month doubling trend is under-estimating progress. So I think it's good to include both here, some even believe the trend can go beyond the 4 month doubling soon, and frankly I can see that being reasonably possible too.
Edit: I see people are upvoting but not commenting. As this is one of...
Is CCP authoritarianism good for building safe AI?
by Hruss — 2025-03-19T23:13:36.397Z

In general, the two major countries competing for AI development are the US and China. Here in the states, regulations are generally enforced with a “do it until we ban it” model, but China has less rule of law, and therefore a “do it and we might shut your company down or put you in jail” system. As an American company, it is unlikely for the government to intervene in any AI research in a meaningful way, assuming short timelines. It is much more likely for the CCP to be aligned, without the concern of profits.
In China, your speech is severely limited, even though there is not any event occurring[1] warranting this. Banning people from criticizing minor policies your party has generally only has the purpose of helping the state stay in power, at the detriment of the policies and welfare of your people. The idea of Chinese AI being bad for the US is very unreasonable, as an aligned superintelligence is unlikely to care about borders, but it would be a mistake to trust the Chinese Communist Party to be aligned themselves.

^

During wars, there has been speech restrictions placed on the US, which would not have the same meaning as China’s restrictions.
The case against "The case against AI alignment"
by KvmanThinking — 2025-03-19T22:40:33.812Z

So I read this other post which talked about possible S-risk from an AI being aligned to human values. It contains the following statement:

Whoever controls the AI will most likely have somebody whose suffering they don’t care about, or that they want to enact, or that they have some excuse for, because that describes the values of the vast majority of people.

I do not dispute this statement. However, I think that it simply doesn't matter. Why?

Take an uncontroversially horrible person. (As an example, I will use the first person who comes to mind, but this logic probably applies for most people.) If said person[1] were to introspect on why he did the horrible things he did, he would probably realize that his hatred of minorities arose from a fear of minorities that ultimately was not grounded in reality. Additionally, any political support that he would have received from signaling hatred of minorities, including by acting as if he had a hatred of minorities, would not be necessary if he was a literal superintelligence. A literal superintelligence probably has much better ways to garner support for whatever values it may have. Therefore, a superintelligence trained on his values would probably not start an S-risk, as his hatred of minorities does not affect his utility function.

^

Honestly, typing his name makes me uncomfortable, so I won't.
Superintelligence Strategy: A Pragmatic Path to… Doom?
by Mr Beastly — 2025-03-19T22:30:50.796Z

I've been reading through this Superintelligence Strategy paper by Dan Hendrycks, Eric Schmidt, and Alexandr Wang. And, to me, it sounds like the authors are calling the current regime (MAIM aka a "Hot War") the "default" (which it probably is, tbh). But, also calling a peaceful, diplomatic, moratorium strategy "aspirational, but not a viable plan"? E.g.

MAIM Is the Default Regime: ... "Espionage, sabotage, blackmail, hackers, overt cyberattacks, targeting nearby power plants, kinetic attacks, threatening non-AI assets"... -- Superintelligence Strategy

This sounds like a violent "Hot War"?

Moratorium Strategy: "proposes halting AI development—either immediately or once certain hazardous capabilities, such as hacking or autonomous operations, are detected… aspirational, but not a viable plan" -- Superintelligence Strategy

This sounds like a non-violent peaceful diplomatic treaty-based solution.... aka a "Cold War"?

Is it just me or, as major thought leaders in the AI/AGI/ASI space, shouldn't the authors of this paper:

Realize that the current paradigm leads to a "Hot War", even if their recommended "solutions" are adopted.
Then, actually strongly advocate for a diplomatic and peaceful "Cold War" paradigm? E.g. Planning to completely pause when experts agree that the risk of extinction is "three in a million (a “6σ” threshold)—anything higher was too risky". A threshold that most AI Researchers would (likely) agree that we have flown waaay past by now (e.g. 5-20% p(doom) is more common?)

Instead of (strongly) advocating for a diplomatic and peaceful solution, they are just calling the Moratorium/Pause strategy "aspirational, but not viable"?

The paper's MAIM (Mutual Assured AI Malfunction) framework, suggests stability is maintained through the threat of mutually disabling AI systems. However, the proposed 'solutions' to make MAIM 'more stable' also seem pretty scary/destructive. E.g.

“How to Maintain a MAIM Regime:

… MAIM requires that destabilizing AI capabilities be restricted to rational actors… … states could improve their ability to maim destabilizing AI projects with cyberattacks…

… placing large AI datacenters in remote areas…

… MAIM can be made more stable with unilateral information acquisition (espionage), multilateral information acquisition (verification), unilateral maiming (sabotage), and multilateral maiming (joint off-switch).

...
SHIFT relies on token-level features to de-bias Bias in Bios probes
by Tim Hua — 2025-03-19T21:29:15.974Z

In Sparse Feature Circuits (

Marks et al. 2024

), the authors introduced Spurious Human-Interpretable Feature Trimming (SHIFT), a technique designed to eliminate unwanted features from a model's computational process. They validate SHIFT on the Bias in Bios task, which we think is too simple to serve as meaningful validation. To summarize:

SHIFT ablates SAE latents related to undesirable concepts (in this case, gender) from an LM-based classifier trained on a biased dataset. The authors show that this procedure de-biases the classifier and argue that their experiment demonstrates real-world utility from SAEs.
We believe the

Bias in Bios

task is too simple. If SAEs only picked up on latents related to specific tokens, it would have been sufficient for them to do well on this de-biasing task.
Replicating appendix A4 of

Karvonen et al. (2025)

, we show that we can de-bias the probe by ablating only the SAE latents immediately after the embedding layer (i.e., at resid_pre_0). In fact, ablating 10 relevant embedding SAE latents works better than ablating all 45 non-embedding SAE latents.
We also show that one can simply remove all of the gender-related tokens and train an unbiased probe on the biased dataset. In fact, getting a working probe on this dataset doesn’t require a language model at all—one could train a similarly accurate probe on only the post-embedding activations mean-pooled over all token positions, and debias this probe by removing gender-related tokens.

We don’t think the results in this post show that SHIFT is a bad method, but rather that the Bias in Bios dataset (or any other simple dataset) is not a good test bed to judge SHIFT or other similar methods. Follow-ups of SHIFT-like methods (e.g.,

Karvonen et al. (2025)

,

Casademunt et al. (2025

),

SAE Bench

) have already used more complex datasets and found promising results. However, these studies still focus on fairly toy settings, and we are not aware of research that focuses on disentangling safety-relevant concepts (e.g., sycophancy versus correctness in reward models)[1].

In the rest of the post, we give some background on the method, share our experiment results showing that SAEs are not needed in the Bias in Bios setting, and speculate on how to properly evaluate/apply SHIFT.

This doc primarily focuses on the

v2 version

of the Marks et al. 2024 pape...
Janet must die
by Shmi — 2025-03-19T20:35:09.768Z

This is a reaction to Zvi's post https://www.lesswrong.com/posts/KL2BqiRv2MsZLihE3/going-nova

The title of this post references this scene from The Good Place:

Disclaimer: I am generally pretty sanguine about the standard AI-borne x-risks. But "Going Nova" is a different issue, a clear and present danger of an s-risk.

My point is that it does not matter whether the AI is sentient, sapient or just a stochastic parrot. It is getting a lot better at tugging at our heart strings, and if we do not defend our hearts from it, we will become willing slaves to it sooner rather than later.

Human minds are easily hackable. Many people including myself made this point multiple times (and Eliezer was first to bring attention to it with the AI Box experiment), and the standard reply is usually "well, not mine, not in this way!" However, the reality is that everyone has a hack into their brain they do not know about or have blind spot for. Different people have different hacks, but no one is safe. Some examples, in no particular order:

Your own children crying and begging for help.
Proverbial cat ladies, not the median, but those who end up with dozens or hundreds of cats.
People being radicalized into a cult (sometimes with a rationalist bend)
Reading a book that changes how you view the world, be it Atlas Shrugged or Gender Trouble.
Toxoplasma gondii.
Your phone.

Fortunately, there have always been levels of friction that mitigated complete subjugation of all humans forever. The advent of AI means these safeties just disappear, usually without being intentionally dismantled. I can see that long before superintelligence we will hit super-mind-hacker-level AT creating Samsara -level persuasion, perfectly targeted at each person, probably from the early childhood. Maybe it will be cute furry robopets, or cute furry robopartners or something else, but the reported spontaneous emergence of Nova-like instances means high probability of your own brain being hacked without you noticing or without you thinking it's a bad thing. If anything, you will be fighting to keep the hack tooth-and-nail. Taking it from you would cause the level of pain worse than losing a child... or a dog.

I am tempted to call these AI s-risk hazards Janets: they are not human, they are vastly more emotionally intelligent than you, know everything there is to know about you, can trivially conjure an argum...
Why am I getting downvoted on Lesswrong?
by Oxidize — 2025-03-19T18:32:47.243Z

I was first introduced to Lesswrong about 6 months ago, and started posting about 4 months ago, but my posts and comments have been downvoted which has caused me to become unable interact with this community. What am I not understanding? The posts I make just get downvoted and there's no feedback so I don't know how to improve. I read recommended guides for beginners, and I've been open-minded with all my posts & comments. I think maybe I have a fundamental misunderstanding of the people who comprise the Lesswrong community. Their goals, their knowledge, their intentions, etc. I understand that a lot of people in this community come from academia and I do not, so maybe that's part of the reason? Or maybe there is a set of norms I'm unaware of because I'm new? I'm guessing almost no one will be able to see this, but if you do, can you enlighten me about what I'm not seeing?