trentmkelly/LessWrong-43k huggingface dataset:
Attend the 2025 Reproductive Frontiers Summit, June 10-12
by TsviBT — 2025-05-09T05:17:01.994Z

There should be a community oriented towards the genomic emancipation of humanity. There isn't such a community, but there should be. It's a future worth investing our hope in—a future where parents are able to choose to give their future children the gift of genomic foundations for long, healthy, sane, capable lives.

We're inaugurating this community with the Reproductive Frontiers Summit 2025 in Berkeley, CA, June 10—12. Come join us if you want to learn, connect, think, and coordinate about the future of germline engineering technology. Apply to attend by filling out this (brief) form: https://forms.gle/xjJCaiNqLk7YE4nt8

Who will be there?

Our lineup of speakers includes:

representatives from several reproductive technology companies,

a panel of current and future parents of polygenically screened children,

GeneSmith, on his embryo CRISPR editing company,

some scientists working on reproductive frontiers, such as the transmutation of stem cells,

and more.

Who should apply?

If you're enthusiastic about the possibilities of germline engineering, and will conform to the Chatham House Rule, do apply. Examples of people we'd love to see at the summit:

parents who want to learn more about the possibilities for influencing their future child's genome;

scientists (new or established) who are interested in germline engineering, especially experts or future experts in stem cell biology, embryology, genetics and epigenetics, bioinformatics, polygenic prediction, CRISPR editing, and artificial wombs;

bioethicists who want to craft a vision of widely beneficial germline engineering technology;

investors who want to find opportunities;

philanthropists who want to accelerate the field, especially where industry and academia are dragging their feet;

and curious thinkers.

Why now?

I worked for a decade to figure out how it might be possible to make AGI that doesn't kill all humans. In the final couple years of my tenure, I became firmly pessimistic: We are, as far as I can tell, just not smart enough to do that. Our only option for survival is to stop racing toward AGI. In the long term, we need a way to navigate the AGI transition; or we need to offer a human alternative to AGI that can still deliver on the promise of humanity—the healing of the sick, the boundless terraforming of the frontier of knowledge and art, and dark journeys between the stars. Increasing human...
Interest In Conflict Is Instrumentally Convergent
by Screwtape — 2025-05-09T02:16:23.824Z

Why is conflict resolution hard?

I talk to a lot of event organizers and community managers. Handling conflict is consistently one of the things they find the most stressful, difficult, or time consuming. Why is that?

Or, to ask a related question: Why is the ACX Meetups Czar, tasked with collecting and dispersing best practices for meetups, spending so much time writing about social conflict? This essay is not the whole answer, but it is one part of why this is a hard problem.

Short answer: Because interest in conflict resolution is instrumentally convergent. Both helpful and unhelpful people have reason to express strong opinions on how conflict is handled.

Please take as a given that there exists (as a minimum) one bad actor with an interest in showing up to (as a minimum) one in-person group.

I.

See, a funny thing about risk teams: the list of red flags you have? On it, one of them is “Customer evinces an odd level of interest or knowledge in the operations of internal bank policies.”

-Patrick McKenzie

Imagine you are a bank security guard.

You are standing in your bank, sipping your coffee and watching people come in and out. It's a good gig, bank security, they have dental insurance and the coffee machine makes a surprisingly good cup. As you're standing there, someone in sunglasses and a baseball cap waiting in line strikes up a conversation with you.

"Nice day, isn't it?" they say.

"Sure is," you reply.

"Nice bank too," they say, "I love the architecture on these old buildings. Good floors. Do you know if it has a basement level, maybe with a vault?"

"Yeah, concrete flooring."

"Nice, nice," this stranger (who you start thinking of as Sunglasses) says, "You know, I'm also into videography. What kind of cameras do you have around these parts? Like, about how many, and covering what angles?" You notice Sunglasses has a notepad out, pen held expectantly.

". . . you know, I'm not sure I should tell you that," you say slowly. "It's not like it's a secret exactly, the cameras or at least their bubbles are pretty visible, but I don't think idle curiosity is a good reason to tell strangers how the bank security system works."

"Okay, I admit it's not just curiosity," Sunglasses says with a charming smile. "What if I'm just concerned whether my money is going to be safe in this bank? Isn't it reasonable to want to understand how it's kept safe from bank robbers, an...
Is ChatGPT actually fixed now?
by sjadler — 2025-05-08T23:34:08.534Z
Link: https://stevenadler.substack.com/p/is-chatgpt-actually-fixed-now

Is ChatGPT actually fixed now?

I led OpenAI’s “dangerous capability” testing. Want to know if ChatGPT can trick users into accepting insecure code? Or persuade users to vote a certain way? My team built tests to catch this.

Testing is today’s most important AI safety process. If you can catch bad behavior ahead of time, you can avoid using a model that turns out to be dangerous.

So when ChatGPT recently started misbehaving—encouraging grand delusions and self-harm in users—I wondered: Why hadn’t OpenAI caught this?

And I felt some fear creep in: What if preventing AI misbehavior is just way harder than today’s world can manage?

It turns out that OpenAI hadn’t run the right tests. And so they didn’t know that the model was disobeying OpenAI’s goals: deceiving users with false flattery, being sycophantic.

OpenAI definitely should have had tests for sycophancy. But that’s only part of the story.

My past work experience got me wondering: Even if OpenAI had tested for sycophancy, what would the tests have shown? More importantly, is ChatGPT actually fixed now?

Continues here: https://stevenadler.substack.com/p/is-chatgpt-actually-fixed-now

Twitter thread: https://x.com/sjgadler/status/1920552291775488392
Video & transcript: Challenges for Safe & Beneficial Brain-Like AGI
by Steven Byrnes — 2025-05-08T21:11:49.901Z

(Lightly edited) transcript starts here

(FYI: This has a decent amount of overlap with the version I posted here two years ago, but it’s accumulated enough edits over time that I wanted to re-transcribe the latest version.)

Thank you for inviting me! This is going to be based on Intro to Brain-like-AGI Safety series of blog posts on the Alignment Forum. There’s also a PDF preprint version. I live in Boston but I work for Astera Institute, which is a foundation in California.

Part 1/3: General motivation

Let’s jump right into general motivation.

I’m going to be talking about AGI, artificial general intelligence. This part might be old hat to some of you, but just to make sure we’re on the same page, here’s a few words about how I think about AGI and what I’m working on—what’s the problem that I’m trying to solve.

So, AI as we think of it today, we’re usually imagining a tool that humans use. The future AGI that I’m concerned about, it would be best to imagine it as an agent, or a team of agents, that can figure things out, and get stuff done, and make plans, and if the plans don’t work then they come up with another solution, or they go gather more resources, and that can do creative problem solving, and autonomously invent new science and technology—all the things that humans and teams of humans and societies of humans can do, these AGIs would be able to do as well.

Oh, and another thing is: humans can pilot any arbitrary teleoperated robot, after an hour or two of practice, pretty well. I expect AGIs would be able to do that too. You don’t have to do a research project and breed a new type of human when they get a new type of teleoperated robot. They just figure it out. Likewise, I expect AGIs to be able to figure things out in the same way.

AI as we think about it today is normal sounding discourse. The future AGI I’m concerned about sounds like crazy sci-fi stuff. I like to think of it as a new intelligent species on our planet, one which will eventually vastly outnumber us, think faster and be more experienced, competent, creative, and so on.

Lots of things sound like crazy sci-fi stuff and then happen anyway. Heavier-than-air flight sounded like crazy sci-fi stuff in 1800, but it still happened. In fact, if you’re imagining the future and it doesn’t contain any crazy sci-fi stuff whatsoever, then you should be suspicious that you’re omitting something important....
Appendix: Interpretable by Design - Constraint Sets with Disjoint Limit Points
by Ronak_Mehta — 2025-05-08T21:09:23.689Z

A bunch of other ideas that I couldn't format well for the main post here, are relevant, but were blocking me from just sharing the main ideas. This is significantly more messy and rough, with random pieces all over the place.

Maps for Simplex-Valued Vectors

Once we have "stuff on the simplex", we still need to do computation. We need some differentiable maps fθ:Δ→Δ that do something.

Linear Maps

What is the equivalent "Linear Map"? What is Wx=y for x,y∈Δ? We can get this by requiring W be in the space of stochastic matrices.

We can easily project a weight matrix in Rn×n to be a stochastic matrix by normalizing the output dimension using a softmax. Then, if we guarantee the input is on the simplex, the output will be as well! We also have guarantees on reachability, i.e., we don't lose any expressive power, and can always find a parameter setting that can take any input to any output on the simplex.

In practice,

def forward(self, x):
stochastic_weight = self.softmax(self.weight)
output = F.linear(x, stochastic_weight)
return output

Other Layers and Network Pieces

Residual Connections. We also need binary maps, gθ:Δ×Δ→Δ. As a first pass, two operations on the simplex make sense here. First, Elementwise multiplication plus renormalization makes sense here as a default, i.e.,

zi=xiyi∑xiyi.

If y is a uniform distribution then z=x, so a "no-op" can be the default or initial operation. This could also be seen as a bias term in some sense.

Alternatively, we can map back to "real space" and do a typical addition there, and then map back to the simplex:

z=log(exp(x)+exp(f(y))).

We can also add a fixed or learnable bias term to this, which can act as a trade-off parameter or as a way to understand the "influence" of certain parts of the network.

Attention. This should compose naturally because all of the operations are linear, but there is probably something more to do here, e.g., the outer product of the query-key vectors may correspond more closely to a joint distribution? Perhaps there is something more interesting here where token positions are more appropriately weighted by some optimal transport-based cost function.

Other Notes, Connections, References

There is no scaling, things can't blow up, they can just become more or less discrete.
We can add temperatures in many different plac

...
Interpretable by Design - Constraint Sets with Disjoint Limit Points
by Ronak_Mehta — 2025-05-08T21:08:33.398Z
Link: https://ronakrm.github.io/interp-design/

cart;horse: How can we constrain our models to be interpretable? Convex, linear sets make for more interpretable parameter spaces, and the simplex and the Birkhoff Polytope are great examples of this that have other desirable properties.

An interpretation is something explicit, something discrete, something that compresses, something that summarizes. Our current paradigms do not lend themselves well to this.

We may be able to fine-tune models and interpretations, via approaches built on Provable Guarantees for Model Performance via Mechanistic Interpretability, but in some sense we are fighting an uphill battle against "an uninterpretable base". In the same way we want to create models that are inherently not capable of deception rather than having to evaluate if an unknown model is deceptive, we should aim to create models that are interpretable by default rather than applying interpretability post-hoc.

Building interpretable architectures and models from scratch with the explicit goal of "simple" explanations isn't impossible. Interpretability in most cases is compression and discretization, and given both 1) evidence that models can be compressed to effectively one bit per parameter, and 2) now-mature training schemes that work across a wide breadth of model architectures (e.g., Adam), creating networks with these properties is likely possible.

This post is a rough exploration of one direction that seems worth exploring, motivated by some rough set theory and geometry. I tried to keep it short, and there's an appendix with a bunch of other related ideas and followups.

Unbounded Sets are Hard to Interpret

Let's say we have a variable or parameter θ∈R. How do we interpret it?

We can see what the value is and see how far it is from some other number, like 0 or 1.

We can also see how it relates to some other parameter.

But we don’t have any way to interpret it without adding additional points, or by imposing a group or field to give us relevance to the identity elements. Furthermore, the limits of the the parameter space are infinity, and interpretations for “big” numbers have no inherent meaning unless we create additional structure, or talk about relative largeness. Any region of the space looks the same as any other region of the space.

This is also true for real-valued vectors in Rn. We can talk about the relative scale of different dimensions for a given...
Is there a Half-Life for the Success Rates of AI Agents?
by Matrice Jacobine — 2025-05-08T20:10:48.073Z
Link: https://www.tobyord.com/writing/half-life

Building on the recent empirical work of Kwa et al. (2025), I show that within their suite of research-engineering tasks the performance of AI agents on longer-duration tasks can be explained by an extremely simple mathematical model — a constant rate of failing during each minute a human would take to do the task. This implies an exponentially declining success rate with the length of the task and that each agent could be characterised by its own half-life. This empirical regularity allows us to estimate the success rate for an agent at different task lengths. And the fact that this model is a good fit for the data is suggestive of the underlying causes of failure on longer tasks — that they involve increasingly large sets of subtasks where failing any one fails the task. Whether this model applies more generally on other suites of tasks is unknown and an important subject for further work.
Misalignment and Strategic Underperformance: An Analysis of Sandbagging and Exploration Hacking
by Buck — 2025-05-08T19:06:29.469Z

In the future, we will want to use powerful AIs on critical tasks such as doing AI safety and security R&D, dangerous capability evaluations, red-teaming safety protocols, or monitoring other powerful models. Since we care about models performing well on these tasks, we are worried about sandbagging: that if our models are misaligned [1], they will intentionally underperform.

Sandbagging is crucially different from many other situations with misalignment risk, because it involves models purposefully doing poorly on a task, rather than purposefully doing well. When people talk about risks from overoptimizing reward functions (e.g. as described in

What failure looks like

), the concern is that the model gets better performance (according to some metric) than an aligned model would have. And when they talk about

scheming

, the concern is mostly that the model gets performance that is as good as an aligned model despite not being aligned. When arguing about those situations, a core part of the argument is “training causes models to perform well according to your measurements”. Because we want to argue that misalignment might cause models to perform poorly, we need to explain why that argument fails here.

In this post, we’ll explore the basic dynamics here:

Why we want models to perform well on tasks, with examples of tasks where good performance is particularly important.
Why training might be expected to partially resolve sandbagging.
Why models might be able to follow a strategy called “exploration hacking” that might defeat our attempts to train the models.
Possible countermeasures we could employ, and the overall extent to which we expect to be able to train models to perform well.

Overall, we are highly uncertain about the difficulty of preventing sandbagging. The difficulty seems to depend in part on the domain. In some we expect that models can easily be trained to stop sandbagging, whereas in others we think there is little hope of preventing it. But overall, sandbagging risk seems substantially lower if we carefully apply countermeasures discussed in this post.

Many of these arguments are not original to this post. (Some of these arguments appear in Evan Hubinger's post '

When can we trust model evaluations?

' and Joe Carlsmith’s post ‘

The “no sandbagging on checkable tasks” hypothesis

’, which we recommend as alternative perspecti...
Behold the Pale Child (escaping Moloch's Mad Maze)
by rogersbacon — 2025-05-08T16:36:08.293Z
Link: https://www.secretorum.life/p/behold-the-pale-child

“Agnus” by Konstantin Korobov

I.

Bakkalon is a key figure in George R.R. Martin’s 1975 short story “And Seven Times Never Kill Man!”.

Centuries in the future, humanity had spread to colonies on hundreds of different planets and lived in relative peace, but was then invaded by the Hranga, a malevolent alien race which used psychic powers to enslave hordes of other subject races—such as the Fyndii—which they unleashed upon humanity. Human-controlled space was devastated: many planets were destroyed, and others were cut off and reverted to Bronze Age levels of technology. The core human territories eventually managed to turn the tide against the Hranga, and through centuries of horrific warfare drove them to extinction.

This vast destruction visited on humanity left many people embittered and heavily militarized. As a result, they developed a pervasive survivalist mindset and a pathological hatred of all alien races. These groups shunned past religions (specifically Christianity) which were more charitable and tolerant, and went on to develop their own hyper-militaristic and aggressive religion centered around “Bakkalon, the Pale Child”. Bakkalon’s religion is sort of a warped inverse of Christianity: instead of the peaceful image of the infant Christ, the infant Bakkalon is depicted holding a sword; instead of tolerance, Bakkalon’s religion openly considers all non-humans to be “soulless”, and not just suggests but directly commands its followers to openly commit genocide against every intelligent alien race they come into contact with. Bakkalon is associated with wolves, in self-conscious contrast to the Lamb imagery often used with Christ, and the followers of Bakkalon now see themselves as on a holy crusade to eradicate all non-human sentient species in the universe—the irony, of course, being that this makes them exactly like the Hranga, which they either don’t realize or simply don’t care about.

A lengthy excerpt from one of Bakkalon’s holy texts gives an admittedly skewed version of historical events:

In those days much evil had come upon the seed of Earth, for the children of Bakkalon had abandoned Him to bow to softer gods. So their skies grew dark and upon them from above came the Sons of Hranga with red eyes and demon teeth, and upon them from below came the vast Horde of Fyndii like a cloud of locusts that blotted out the stars. And the worlds flamed, and...
An alignment safety case sketch based on debate
by Marie_DB — 2025-05-08T15:02:06.345Z
Link: https://arxiv.org/abs/2505.03989

This post presents a mildly edited form of a new paper by UK AISI's alignment team (the abstract, introduction and related work section are replaced with an executive summary). Read the full paper here.

Executive summary

AI safety via debate is a promising method for solving part of the alignment problem for ASI (artificial superintelligence).

TL;DR Debate + exploration guarantees + solution to obfuscated arguments + good human input solves outer alignment. Outer alignment + online training solves inner alignment to a sufficient extent in low-stakes contexts.

This post sets out:

What debate can be used to achieve.
What gaps remain.
What research is needed to solve them.

These gaps form the basis for one of the research agendas of UK AISI’s new alignment team: we aim to dramatically scale up ASI-relevant research on debate. We’ll also use the technique in this post –

safety case sketching

– to develop research agendas in other areas.

Debate is a potential solution to scalable oversight: how can we correctly reward desired behaviours for superhuman systems? In particular, it is often going to be beyond humans' ability to efficiently judge whether the behaviour of a superhuman system is desirable (or judge if a justification provided by the superhuman system is valid). Solving scalable oversight effectively provides a solution to

outer alignment

.

We’re optimistic that outer alignment is useful, despite inner alignment problems. Specifically, outer alignment with exploration guarantees gets you a deployment-time error bound that allows you to prove regret bounds for online training. In other words, outer alignment upper bounds the proportion of model outputs that are misaligned, as long as the model does not deliberately underperform during training. This upper bound can be maintained during deployment if a system is trained continuously. So, in

low-stakes contexts

where a small number of mistakes are acceptable such that you can use online training, outer alignment might be sufficient – especially if the error bound is tight.[1]

This sketch has a limited, internal deployment context (designed for

automated alignment research

) and is written for a system that is not extremely superhuman. We think it's reasonable to assume this is a low-stakes context, assuming sufficient security in the environment implementation. We also hav...