trentmkelly/LessWrong-43k huggingface dataset:
AE Studio is hiring! 
by AE Studio — 2025-04-21T20:35:55.818Z

AE Studio is Hiring: Data Scientists & ML Engineers for AI Alignment Work

Location: Remote or Los Angeles (in-person strongly encouraged)
Type: Full-time
Compensation: Competitive salary + meaningful equity in client and Skunkworks ventures

Who We Are

AE Studio is an LA-based tech consultancy focused on increasing human agency, primarily by making the imminent AGI future go well. Our team consists of the best developers, data scientists, researchers, and founders. We do all sorts of projects, always of the quality that makes our clients sing our praises.

We reinvest those client work profits into our

promising research on AI alignment

and our ambitious internal skunkworks projects. We previously

sold one of our skunkworks

for millions of dollars.

We have

made a name

for ourselves in cutting-edge

brain computer interface

(BCI) R&D, and after working on this for the past two years, we have made a name for ourselves in

research and policy efforts

on AI alignment. We want to optimize for human agency, if you feel similarly, please apply to support our efforts.

What We’re Doing in Alignment

We’re applying our "

neglected approaches

" strategy—previously validated in BCI—to AI alignment. This means backing underexplored but promising ideas in both technical research and policy. Some examples:

Investigating self-other overlap in agent representations
Conducting feature steering using Sparse Autoencoders
Looking into information loss with out of distribution data
Working with alignment-focused startups (e.g., Goodfire AI)
Exploring policy interventions, whistleblower protections, and community health

You may have read some of our work here before but for a refresher, feel free to go to our LessWrong profile and get caught up on our thought pieces and research.

Interested in more information about what we’re up to? See a summary of our work here:

https://ae.studio/ai-alignment

About You

Passionate about AI alignment and optimistic about humanity’s future with AI
Experienced in data science and ML, especially with deep learning (CV, NLP, or LLMs)
Fluent in Python and familiar with calling model APIs (REST or client libs)
Love using AI to automate everything and move fast like a startup
Proven ability to run projects end-to-end and break down complex problems
Comfortable working autonomously and explaining technical ideas clearly to any a

...
$500 Bounty Problem: Are (Approximately) Deterministic Natural Latents All You Need?
by johnswentworth — 2025-04-21T20:19:30.808Z

Our posts on natural latents have involved two distinct definitions, which we call "stochastic" and "deterministic" natural latents. We conjecture that, whenever there exists a stochastic natural latent (to within some approximation), there also exists a deterministic natural latent (to within a comparable approximation). We are offering a $500 bounty to prove this conjecture.

Some Intuition From The Exact Case

In the exact case, in order for a natural latent to exist over random variables X1,X2, the distribution has to look roughly like this:

Each value of X1 and each value of X2 occurs in only one "block", and within the "blocks", X1 and X2 are independent. In that case, we can take the (exact) natural latent to be a block label.

Notably, that block label is a deterministic function of X.

However, we can also construct other natural latents for this system: we simply append some independent random noise to the block label. That natural latent is not a deterministic function of X; it's a "stochastic" natural latent.

In the exact case, if a stochastic natural latent exists, then the distribution must have the form pictured above, and therefore the block label is a deterministic natural latent. In other words: in the exact case, if a stochastic natural latent exists, then a deterministic natural latent also exists. The goal of the $500 bounty is to prove that this still holds in the approximate case.

Approximation Adds Qualitatively New Behavior

If you want to tackle the bounty problem, you should probably consider a distribution like this:

😏

Distributions like this can have approximate natural latents, while being qualitatively different from the exact picture. A concrete example is the biased die: X1 and X2 are each 500 rolls of a biased die of unknown bias (with some reasonable prior on the bias). The bias itself will typically be an approximate stochastic natural latent, but the lower-order bits of the bias are not approximately deterministic given X (i.e. they have high entropy given X).

The Problem

"Stochastic" Natural Latents

Stochastic natural latents were introduced in the original Natural Latents post. Any latent Λ over random variables X1,X2 is defined to be a stochastic natural latent when it satisfies these diagrams:

... and Λ is an approximate stochastic natural la...
More Than Just A, T, C, and G: Screening for Hidden Dangers in DNA Sequences
by sgd — 2025-04-21T20:12:21.522Z

More Than Just A, T, C, and G: Screening for Hidden Dangers in DNA Sequences

Introduction

Synthetic biology has revolutionised the way modern biosciences are now done, impacting every field within the subject. By engineering organisms in predictable ways, it is possible to create novel biological solutions using genetic components from other species, and those developed in other ways, to give specific characteristics to the species of interest. Recent developments in nucleic acid synthesis allow sequences (such as DNA and RNA) to be produced ‘to order’, it is now possible to engineer a wide variety of species in ways previously not thought to be possible. Synthetic biology has become a growing field, and is frequently described as being at the inflection point computing was in the 1950s.

The advent of machine learning and artificial intelligence will accelerate this work, allowing DNA sequences to be computationally engineered ‘in silico’ and then synthesised. By automating highly repetitive and time consuming lab tasks, organisms can be engineered at a faster pace and more significantly.

With all good technologies come drawbacks. One issue frequently raised with synthetic biology is dual-use, where a technology developed for good (for example, making wheat more drought resistant) can also be applied to a malicious use (such as developing a more virulent form of flu). Bad actors could misuse the technology of today to develop more virulent versions of pathogens like smallpox, malaria, or measles, evading existing immunity, spreading more quickly, and inflicting significant economic and societal harm. Or with the technology of tomorrow, completely novel, never before seen pathogens could be developed, wreaking havoc on society.

Combating the risks of synthetic biology will be an important process in the coming years, helping to make the field safe as more people get trained in the techniques, and general understanding becomes more widespread.

One way of reducing the risk of synthetic biology is by screening nucleic acid orders, and this is what I’ll focus on here. Scientists can order synthetic nucleic acids, for example that have been engineered with computational techniques, and implement these in their engineered organisms. There are contractors that will send you DNA from a sequence you provide, or by using a piece of equipment called a ‘benchtop synthesiser’ f...
The US Executive vs Supreme Court Deportations Clash
by NunoSempere — 2025-04-21T19:56:03.711Z
Link: https://blog.sentinel-team.org/p/the-us-executive-vs-supreme-court

Forecaster perspectives

Sentinel forecasters in aggregate assess as “83% true” (65% to 100%) the statement that the Trump administration has disobeyed the Supreme Court so far. On the one hand, the Trump administration has done nothing to “facilitate” bringing Abrego García back to the US, and this White House’s tweet shown below clearly indicates that it won’t try to; on the other hand, it could yet still do so when the current news cycle subsides. On April 17th, forecasters estimated a 66% chance (50% to ~100%) that Abrego García will not be brought back to the US within the next 60 days (by June 17).

Data on the number of deportations in past administrations are available but can be difficult to compare between administrations; some up-to-date sources include refusals at the border due to Covid (“Title 42 expulsions”) under Biden, but other data have been discontinued. Perhaps a good comparison might be to the years 2012-2015, Obama’s second term, which saw a total of about 1.5M immigrants removed over all four years. The Trump administration has reportedly set a goal of removing 1M people in its first year. As of April 19, 3.6 months into Trump’s second term, the administration has deported 0.11M people. Removing 1M people by the end of 2025 would require a removal rate that is almost four times higher than the rate to date.

Our forecasters estimate a 32% (range, 15% to 70%) probability that the Trump administration will deport over 1M immigrants this calendar year. The administration has the intent, but obstacles include the courts, logistics, incompetence, and possible protection in sanctuary cities, although protection of immigrants by cities could lead to standoffs between local and federal officials. ICE’s recently granted access to IRS data on undocumented immigrants would likely make deportations easier. Paradoxically, those who pay taxes and have tried to work within the system are likely to be at greatest risk of deportation. DOGE personnel are also reportedly constructing a database linking Department of Homeland Security, Social Security Administration and IRS data on undocumented immigrants together, to facilitate deportations. If the Trump administration does deport at least 1M immigrants this year, then we think there will likely be a false positive rate of 0.3% to 5% regarding people mistakenly deported.

Our forecasters estimate a 14% (range, ...
Podcast on “AI tools for existential security” — transcript
by Lizka — 2025-04-21T19:26:07.518Z
Link: https://pnc.st/s/forecast/30c70e39/ai-tools-for-existential-security-with-lizka-vaintrob-

About two months ago, @finm and I recorded a discussion about my now-published piece on “AI tools for Existential Security”[1] (for the podcast Fin has been running at Forethought). I’d never done something like this before, and liked the result a lot more than I thought I would![2]

I'm sharing a quick transcript here in case anyone appreciates section headers, hyperlinks, etc.. Please note that this is very rough; it's an unpolished, not-double-checked “MVP” edit of an AI-generated transcript, and will definitely have mistakes.[3]

Here’s the episode itself; you should also be able to find it on most podcast apps. The discussion loosely follows the structure of the piece it focuses on.

My thinking on some of what we discuss here has changed a bit, but I'd still say most of this (sometimes with somewhat different emphasis, framings, etc.) and I wanted to quickly get this out there. (I plan to share more on this broad topic soon.)

The other episodes of “ForeCast” that currently exist are:

Will AI R&D Automation Cause a Software Intelligence Explosion? (with Tom Davidson)
Preparing for the Intelligence Explosion (with Will MacAskill)
Intelsat as a Model for International AGI Governance (with Rose Hadshar)

I’m behind on listening to my podcast queue, even for episodes recorded by my colleagues,[4] and am embarrassed to say that I haven’t yet listened to most of these so can’t say much more!

Transcript

Elevator pitch & how this relates to d/acc and def/acc

Fin

Okay, I am Fin and I'm speaking with Lizka Vaintrob about a memo that you’ve co-written with Owen Cotton-Barratt, on AI applications for existential security.

So what is the elevator pitch for this?

Lizka

The basic idea is:

“What AI is doing” is increasingly shaping the world. But AI is not one big thing, and much like we have differential technological progress — which is the idea that you can bring forward positive technologies and push back risk increasing technologies — we might be able to shape the character of AI that is available to us, and the exact applications that exist in the world in a way to help us navigate the coming challenges.

I actually think that the opportunity here is fairly big.

You can cut this space up in various ways. You can talk about shaping the paradigms of AI development. So that's like, maybe you want to try to help more inter...
Implications for the likelihood of human extinction from the recent discovery of possible microbial life
by Mvolz — 2025-04-21T19:15:15.766Z

You might have seen the recent news that there are signatures that suggest the planet K2 18-b might be the home of microbial life.

The Fermi Paradox asks: if space is so big, why haven't the aliens contacted us yet?

One possible resolution of the Fermi paradox is simply that the evolution of life itself is disproportionately improbable, and that's why we haven't encountered any evidence of life thus far.

Each step that reduces the chance of extraterrestrial life leaving evidence we can find is a filter. Hanson posits there is one single step, termed the Great Filter, that has the largest impact on this. Whether this filter is ahead of us or behind us has implications for our future.
If it turns out microbial life did indeed separately evolve on K2 18-b (some say such claims are "premature"), its proximity to us gives us a rough estimate for the probability of life evolving that is relatively high, and suggests the evolution of microbial life is not itself the Great Filter.
This means that other explanations for the Fermi paradox increase in probability, even if there is no single one great filter. A few of these are behind us; one is that evolution of complex life is relatively very improbable, or that the evolution of life that can use technology is relatively very improbable.

But it also increases the likelihood that there is a Great Filter ahead of us, not behind us, and that life capable of technological progress just doesn't tend to last terribly long.

Of course, maybe there is no microbial life on K2 18-b after all.
Key event tracker for AI2027
by MarkelKori — 2025-04-21T19:02:38.928Z

An article you've all probably heard of - AI2027 - has caused me deep surprise. And a desire to do something that can help us not to find ourselves in a hypothetical “acceleration” scenario, no matter how hypothetical it may be.

So, the article describes a number of events that the authors believe are very likely to happen in the next two and a half years, including:
- The emergence of autonomous agents;
- Building the world's largest data centers (although this is already happening);
- China staging a blockade or invasion of Taiwan;
- Scale theft/leaks, etc.

I think it would be a good idea to create a small app on phones or web interface that tracks the events from this article. This app, using a system of keywords or heuristics, would regularly check news from a few dozen credible sources, and then if it matched the predictions from the article, it would alert the user.

This tool would help us understand what phase we are in and how fast the scenario is moving. It will be especially useful for AI Safety researchers, journalists and most importantly, ordinary people who want to stay informed.
Load Bearing Magic
by winstonBosan — 2025-04-21T18:53:25.014Z

In the days of yore, you do not invoke magic, but to become for a moment the faucet it pours from into this world. In the good tsarist years, the scaffold known to us witches today as the fundamental layer of magic was not yet invented. Without these precautions, our predecessors practised a greater degree of freedom in how they perished.

Some lit themselves on fire eternal, seared their shape into the air like some remnant of a flesh statue lighting the way towards folly. Others still lie flattened to the atoms, scattered to the winds with their mind still intact. A few have achieved immortality but as some forgotten bricks of flesh trapped under sediments and dead matters; Only a few had obtained the ultimate prize.

All witches today owe a debt of gratitude to those that have achieved that prize. Gullveig-then-Heiðr comes to mind, who consoled kings and gods on immortality. Others, unlike Heiðr who performs the dance of seidr that re-weaved fate by her lonesome, have sought a more egalitarian means to order. Serving not demiurges, they wanted to disenchant magic and bring them to heel. To decompose magic is to be a class traitor of the divinely chosen few and to risk the wrath of the wicked and the righteous. Breaking them down bit by bit, syllable by syllable so that it may become simple. So simple that anyone can reweave fate with their fellows.

The Great Lord Yang-of-Shang the Lawchanger may be better known for his staunch defence of the equality of law and beheading the king's relatives. But he beheaded more than just princes - all needless abstractions are his enemy. By the time he was finished with his works, Yang‑of‑Shang had beheaded almost everything that could not be cleanly named and purposefully used.

He razored secret glyphs away from the public ones, tore prefixes from nouns, and balanced each symbol’s weight upon a scale of lead millet seeds. When scribes complained that the lexicon would no longer breathe, he answered, “Breathing is an accident. Let us keep only what is essential.” He began by chiselling every known charm onto a cypress tree the size of a threshing floor, then reduced it to the size of a mill, then to the size of a measly plank - and then he splintered it to pieces.

Each shrapnel bore a single stroke — be it a hook of ink, a turn of a brush, a half‑rung in a ladder of thunder where the moon climbs. He showed that any spell, howe...
The Uses of Complacency
by sarahconstantin — 2025-04-21T18:50:02.725Z

Midjourney

Our Culture Expects Self-Justification

I really like David Chapman’s explication of what he calls “reasonableness” and “accountability.”

At least in the culture he and I live in, one is constantly called to account for one’s behavior. At any moment, one may be asked “what are you doing?” or “why did you do that?” And one is expected to provide a reasonable answer.

What’s a reasonable answer?

It’s not a rigorous, beyond-a-shadow-of-a-doubt proof that at this very moment one is engaging in “optimal” or “morally correct” or “maximally virtuous” behavior. That would indeed be an unfair expectation! Nobody could possibly provide such a justification on demand.

What people actually expect is a more-or-less plausible-sounding account that makes your actions sound understandable, relatable, and okay.

You have a lot of leeway to present yourself in a favorable light. Not infinite leeway — if you’re caught shoplifting, it probably won’t be regarded well, no matter what you say — but within a given range of acceptable behavior, people will generally take your word that you’re behaving reasonably, if you can give a few seconds’ worth of calm, appropriate-sounding patter about it.

If you’re nervous, flustered, furtive, or resentful — if you act like you’ve been “caught” slacking, or like it’s embarrassing that you’re trying and failing to do something — you will have a much worse time socially.

There are a lot of echoes of this in different contexts.

Asking for favors

A (perhaps unreplicable) famous 1977 study found that people were more inclined to let a research assistant cut in front of them at the copy machine if they gave an excuse starting with “Because.”

“Because I’m in a rush” (an actual, if thin, justification) and “because I have to make some copies” (a nonsensical/trivial statement) worked equally well!

Common sense confirms that something like this is going on when we ask favors in everyday life.

You are more likely to get a favor granted if you have a “reason” why you particularly need or deserve it, or why it’s particularly easy to grant, or something — but people don’t apply that much critical scrutiny to the quality of your reasons. Maybe, if the request is minor enough, the content of your “reason” is literally irrelevant, as in the copier experiment!

In other cases it may not be that extreme, but you generally don’t need to justify your “reason” that th...
Feature-Based Analysis of Safety-Relevant Multi-Agent Behavior
by Maria Kapros — 2025-04-21T18:12:13.548Z

Introduction

TL;DR

Today’s AI systems are becoming increasingly agentic and interconnected, giving rise to a future of multi-agent (MA) systems (MAS). It is believed that this will introduce unique risks and thus require novel safety approaches.

Current research evaluating and steering MAS is focused on behavior alone i.e inputs and outputs. However, we hypothesize that internal-based techniques might provide higher signal in certain settings. To study this hypothesis, we sought to answer the following questions in multi-LLM agent Iterated Prisoner’s Dilemma (PD):

Can we find meaningful safety-relevant internal representations?
Do mechanistic interpretability (MI) techniques enable more robust steering compared to baselines e.g. prompting?

We observe that PD agents systematically develop internal representations associated with deception. If these results were to generalize to real-world settings, MI tools could be used to enhance monitoring systems in dynamic MA settings.

The objectives for this post are:

Outline our approach and results during the

Women in AI Safety Hackathon

in a more accessible format.
Present additional experiments run to address the feedback received
Sketch a potential project plan for future work

Background and Motivation

From Single- to Multi-Agent AI Safety. In addition to interventions that reduce risks from individual AI systems we might need safety measures that focus on the interactions among AI agents, between them, humans and surrounding digital or physical infrastructure. Such measures could be classified into the following broad clusters:

Monitoring: Methods and infrastructure to evaluate MA risks

Behavioral (based on inputs and outputs) or mechanistic (based on model internals) evaluations to:

Evaluate cooperative capabilities and propensities
Test for dangerous capabilities e.g manipulation, overriding safeguards

Oversight layers

: deployment-time infrastructure that enables actors (humans or digital systems) to detect and intervene upon unintended agent behavior

Governance:

Field-building programs to support research on MA risks
Restrictions on development and deployment

AI tools that help humans resolve major cooperation challenges

Cooperative AI: research and implementation to improve the cooperative and collaborative capabilities of AI systems

Adaptive mechanism design

Peer incentivization methods
Secure communication protocols

Multi-Agent...