trentmkelly/LessWrong-43k huggingface dataset:
How Language Models Understand Nullability
by Anish Tondwalkar — 2025-03-11T15:57:28.686Z
Link: https://dmodel.ai/nullability/

TL;DR Large language models have demonstrated an emergent ability to write code, but this ability requires an internal representation of program semantics that is little understood. Recent interpretability work has demonstrated that it is possible to extract internal representations of natural language concepts, raising the possibility that similar techniques could be used to extract program semantics concepts. In this work, we study how large language models represent the nullability of program values. We measure how well models of various sizes at various training checkpoints complete programs that use nullable values, and then extract an internal representation of nullability.

Introduction

The last five years have shown us that large language models, like ChatGPT, Claude, and DeepSeek, can effectively write programs in many domains. This is an impressive capability, given that writing programs involves having a formal understanding of program semantics. But though we know that these large models understand programs to an extent, we still don’t know many things about these models’ understanding. We don’t know where they have deep understanding and where they use heuristic reasoning, how they represents program knowledge, and what kinds of situations will challenge their capabilities.

Fortunately, recent work in model interpretability and representation engineering has produced promising results which give hope towards understanding more and more of the internal thought processes of LLMs. Here at dmodel​ , we can think of no better place to apply these new techniques than formal methods, where there are many abstract properties that can be extracted with static analysis. The vast work done in programming language theory over the past hundred years provides many tools for scaling an understanding of the internal thought processes of language models as they write code.

In that spirit, we wanted to start with a simple property that comes up in every programming language, nullability. Nullable values are represented differently across languages; as null pointers in C or C++, with explicit Option types in Rust, and with special nil or None values in dynamic languages like Javascript, Lisp, or Python. In every case, understanding where values can be nullable is necessary for writing even basic code, and misunderstanding where they are nullable can often be a source o...
Forethought: a new AI macrostrategy group
by Max Dalton — 2025-03-11T15:39:25.086Z

Forethought[1] is a new AI macrostrategy research group cofounded by Max Dalton, Will MacAskill, Tom Davidson, and Amrit Sidhu-Brar.

We are trying to figure out how to navigate the (potentially rapid) transition to a world with superintelligent AI systems. We aim to tackle the most important questions we can find, unrestricted by the current Overton window.

More details on our website.

Why we exist

We think that AGI might come soon (say, modal timelines to mostly-automated AI R&D in the next 2-8 years), and might significantly accelerate technological progress, leading to many different challenges. We don’t yet have a good understanding of what this change might look like or how to navigate it. Society is not prepared.

Moreover, we want the world to not just avoid catastrophe: we want to reach a really great future. We think about what this might be like (incorporating moral uncertainty), and what we can do, now, to build towards a good future.

Like all projects, this started out with a plethora of Google docs. We ran a series of seminars to explore the ideas further, and that cascaded into an organization.

Research

Research agendas

We are currently pursuing the following perspectives:

Preparing for the intelligence explosion: If AI drives explosive growth there will be an enormous number of challenges we have to face. In addition to misalignment risk and biorisk, this potentially includes: how to govern the development of new weapons of mass destruction; what rights to give digital beings; how to govern an automated military; and how to avoid dictatorship or authoritarianism.
Achieving a near-best future: Most explicitly longtermist work to date has been focused on avoiding existential catastrophe, but achieving a near-best future might be even more important. Research avenues here include mapping out what a desirable “viatopia” would look like (i.e. a state of the world which is very likely to lead to a very good future), figuring out how space resources should be allocated, and addressing issues around AI epistemics and persuasion.

We tend to think that many non-alignment areas of work are particularly neglected.

However, we are not confident that these are the best frames for this work, and we are keen to work with people who are pursuing their own agendas.

Recent work

Today we’re also launching “Preparing for the Intelligence Explosion”, which ...
Preparing for the Intelligence Explosion
by fin — 2025-03-11T15:38:29.524Z
Link: https://www.forethought.org/research/preparing-for-the-intelligence-explosion

This is a linkpost for a new paper called

Preparing for the Intelligence Explosion

, by Will MacAskill and Fin Moorhouse. It sets the high-level agenda for the sort of work that

Forethought

is likely to focus on.

Some of the areas in the paper that we expect to be of most interest to EA Forum or LessWrong readers are:

Section 3 finds that even without a software feedback loop (i.e. “recursive self-improvement”), even if scaling of compute completely stops in the near term, and even if the rate of algorithmic efficiency improvements slow, then we should still expect very rapid technological development — e.g. a century’s worth of progress in a decade — once AI meaningfully substitutes for human researchers.
A presentation, in section 4, of the sheer range of challenges that an intelligence explosion would pose, going well beyond the “standard” focuses of AI takeover risk and biorisk.
Discussion, in section 5, of when we can and can’t use the strategy of just waiting until we have aligned superintelligence and relying on it to solve some problem.
An overview, in section 6, of what we can do, today, to prepare for this range of challenges.

Here’s the abstract:

AI that can accelerate research could drive a century of technological progress over just a few years. During such a period, new technological or political developments will raise consequential and hard-to-reverse decisions, in rapid succession. We call these developments grand challenges.

These challenges include new weapons of mass destruction, AI-enabled autocracies, races to grab offworld resources, and digital beings worthy of moral consideration, as well as opportunities to dramatically improve quality of life and collective decision-making.

We argue that these challenges cannot always be delegated to future AI systems, and suggest things we can do today to meaningfully improve our prospects. AGI preparedness is therefore not just about ensuring that advanced AI systems are aligned: we should be preparing, now, for the disorienting range of developments an intelligence explosion would bring.
stop solving problems that have already been solved
by dhruvmethi — 2025-03-11T15:30:41.896Z

Crossposted from here.

I attended a collegiate Bollywood-Fusion dance competition a couple of weekends ago. This setting was nothing new to me; I competed at many competitions back when I was in college, and after graduating college, I spent a couple of years volunteering with a group that organized a large competition and assisted other competition-organizing groups with their logistics and planning. Basically, through my experiences, I had a high degree of familiarity with how these events typically run.

In my experience, the challenges associated with organizing these collegiate dance competitions are what I would call solved problems: problems for which already existing solutions have been adequately demonstrated. Each competition's core outcomes and processes are largely the same, and the organizing group’s responsibilities and tasks are generally static year after year. Even though Bollywood-Fusion dance is relatively new in the United States, there have probably been over 500 of these competitions run across the country over the last 10-15 years.

You’d expect each of these competitions to run like well-oiled machines by now, but I was really surprised to see that many of the things I viewed as solved problems…weren’t.

The emceeing and awards presentation was awkward. Teams were taking flagrantly unnecessary amounts of time setting up for their performances. Transitions between sets weren’t seamless. All this contributed to an experience that at times felt disjointed and clunky for the audience. And the worst part? The solutions to these problems weren’t secrets! They were widely known and well-documented.

I had no idea why this was happening, but it drove me insane enough that I decided I never wanted to run into this issue again. So I built myself a framework — a playbook — designed to prevent myself (or any team I'm part of) from falling into the same trap.

step 1: throw away roles, responsibilities, and titles — define problems instead

One of the biggest mistakes I’ve seen teams make is that they over-index on defining roles and responsibilities instead of focusing on the actual problems they need to solve. It’s so strange; very few people question why they dream more of holding a certain title or attaining a certain set of responsibilities than of the problems they’ll solve or how they want to make things better. Generally, it’s like shuffling deck chair...
AI Control May Increase Existential Risk
by Jan_Kulveit — 2025-03-11T14:30:05.972Z

Epistemic status: The following isn't an airtight argument, but mostly a guess how things play out.
Consider two broad possibilities:

I. In worlds where we are doing reasonably well on alignment, AI control agenda does not have much impact.

II. In worlds where we are failing at alignment, AI control may primarily shift probability mass away from "moderately large warning shots" and towards "ineffective warning shots" and "existential catastrophe, full takeover".

The key heuristic is that the global system already has various mechanisms and feedback loops that resist takeover by a single agent (i.e. it is not easy to overthrow the Chinese government). In most cases where AI control would stop an unaligned AI, the counterfactual is that broader civilizational resistance would have stopped it anyway, but with the important side effect of a moderately-sized warning shot.

I expect moderately sized warning shots to increase the chances humanity as a whole takes serious actions and, for example, steps up efforts to align the frontier labs.

I am skeptical that incidents stopped by AI control would lead to meaningful change. Sharing details of such an event with proper framing could pose existential risk, but for the lab involved. In practice, I anticipate vague, sanitized communications along the lines of "our safety systems performed as designed, preventing bad things". Without clear, compelling evidence of the severity of the averted threat, these incidents are unlikely to catalyze serious action. The incentives for labs to downplay and obscure such events will be strong.

There are additional factors to consider, like AI control likely moves some resources away from alignment, but I don't think this is the dominant effect.

Note that this isn't a general argument against boxing, e.g. boxes based more on formal methods or theory have better chance to generalize.

Typical counter-arguments to this line of reasoning claim seem to be:

We will extract useful "automated alignment" work from the unaligned AIs inside of the control scheme. I'm sceptical: will cover this in a separate post
Isn't this general counter-argument to alignment research as well? In my view not, details matter: different strains of alignment research have different generalization profiles.

Note: this text lived in a draft form before John Wentworth posted his Case Against AI Control Resea...
When is it Better to Train on the Alignment Proxy?
by dil-leik-og — 2025-03-11T13:35:51.152Z

This is a response to Matt's earlier post. If you see "a large mixture of alignment proxies" when you look at a standard loss function, my post might save you from drawing silly conclusions from the earlier post. If you parse the world into non-overlapping magisteria of "validation losses" and "training losses", then you should skim this but I didn't write it for you.

TLDR:

Proxies for alignment are numerous and varied, with many already existing within your training objective (very few things have exactly zero correlation with alignment).

Different proxies serve different purposes: some are better held in reserve, some are better used in training, and some are worth anti-training on. What you don't train on determines:

Your ability to measure alignment
Your cost of retraining
Your odds of ending up with a misaligned model

The claim that you are always better off not training on a proxy of alignment (when restricted to cases where pass-fail misaligned models are "close in weight space") is false.

The opposite claim (that training on the validator is always better) is also false.

The optimal approach depends on the properties of the proxy and your goals—whether you're trying to learn about the baseline alignment of your ideas or achieve the best outcome given your training budget.

What you should do with a good proxy and a bunch of retries is to learn which of your training ideas are bad and should be thrown out. This dominates rejection sampling.
It is possible that SGD would hack your proxy immediately AND your fully trained model would not hack your proxy in its forward pass/CoT, but it is unlikely when your model is very smart. Even in this case, doing rejection sampling earlier in training (when your model is weaker), probably dominates doing rejection sampling at the very end.

Informal:

Quality of the proxy:

It seems intuitive that training on a proxy for alignment makes it a worse proxy for alignment. Any misaligned model trained on a given proxy will likely pass that proxy. If your goal is purely to measure a model's alignment (rather than produce an aligned model), avoid explicitly maximizing your proxy during training. You should also try to remove implicit maximization of your proxy to preserve its measurement value.

Several factors can degrade a proxy's ability to distinguish between aligned and misaligned models:

Training that instills sticky values
Trainin

...
A different take on the Musk v OpenAI preliminary injunction order
by TFD — 2025-03-11T12:46:23.497Z
Link: https://www.thefloatingdroid.com/a-different-take-on-the-musk-v-openai-preliminary-injunction-order/

The judge in the Musk v OpenAI[1] case out of the northern district of California has issued an order on Musk's motion for preliminary injunction, which asked for an order temporarily stopping OpenAI from taking further steps towards its possible conversion from nonprofit to for-profit control. As might be expected for such high profile litigants, this has received quite a bit of attention in the media, pointing out that Musk's request for a preliminary injunction was denied. However, several commentators who I perceive to be interested in AI safety have jumped in to say "not so fast!". They argue that while its true that Musk was denied a preliminary injunction, digging deeper reveals the situation to be much less favorable to OpenAI then it may seem. Here's how Garrison Lovely (in the second link above) puts it:

In a narrow sense, Musk did "lose." However, as Judge Rogers notes, the bar for a preliminary injunction is extremely high. To surpass it, both the facts of the case and the legal questions involved have to clearly point in the same direction.

Therefore, it would have been extremely surprising if the judge granted the injunction. She did, however, stop just short of it.

Elaborating on the reasoning, Garrison identifies the issue as one of "standing". Rob Wiblin, the other commentator linked above, describes in a podcast interview with nonprofit legal expert Rose Chan Loui this standing issue as follows:

So in a sense, that’s a slightly boring technical legal question. Maybe the juicier interesting thing, and the even bigger update, was having demonstrated that he can bring the case of, is OpenAI doing something objectionable? Is it violating the public’s trust? And is it violating the agreement basically with Musk to use his charitable giving to pursue the charitable mission? And there the judge has seemed very sympathetic to Musk, basically.

Lovely again on the deeper implications that we can infer from this order:

So why does this ruling matter? Well, while Judge Rogers found Musk's standing uncertain at this preliminary stage, she went out of her way to signal that the core claim — that OpenAI's conversion violates its charitable purpose — could have merit if properly brought before the court.

I think part of the excitement here on the part of AI safety folks is that if they are right, this could be an important positive development and opportunit...
Do reasoning models use their scratchpad like we do? Evidence from distilling paraphrases
by Fabien Roger — 2025-03-11T11:52:38.994Z
Link: https://alignment.anthropic.com/2025/distill-paraphrases/

TL;DR: We provide some evidence that Claude 3.7 Sonnet doesn’t encode hidden reasoning in its scratchpad by showing that training it to use paraphrased versions of the scratchpads does not degrade performance.

The scratchpads from reasoning models look human understandable: when reasoning about a math problem, reasoning models consider intermediate steps similar to the ones I would use, backtrack and double-check their work as I would.[1]

But even if scratchpads look like they perform human-like reasoning, scratchpads might improve performance through some less human-understandable mechanisms. One particularly worrying possibility is that models could encode additional reasoning in syntax of the text (e.g. encoding a bit in using a bulleted vs a numbered list, and then using this bit later in the scratchpad). This is sometimes called

encoded reasoning

or

Chain-of-Thought steganography

. If LLMs learned how to use encoded reasoning during RL, they might be able to use it in deployment to reason about how to cause catastrophes while making their reasoning look like benign reasoning, which might increase their ability to scheme and cause catastrophes compared to worlds where they can only figure out how to cause a catastrophe in a single forward pass.

To evaluate whether reasoning models rely on encoded reasoning, we can try to remove or randomize the syntax of the scratchpads by paraphrasing them, and then make the model use these paraphrased scratchpads. To do so, we take 10,000 correct math-related scratchpads from an early checkpoint of Claude 3.7 Sonnet, paraphrase them using Claude 3.5 Sonnet, and then fine-tune the base model of Claude 3.7 Sonnet to produce the paraphrased reasoning (i.e. we distill paraphrased scratchpads into the base model). If the performance of this model is close to the performance of the same base model fine-tuned on the pre-paraphrase scratchpad, then this strongly indicates that the model is not relying on the syntax to get the right answer to math problems during RL,[2][3] while if performance is weaker, then this means that learning the paraphrased scratchpads well enough that they result in high performance is harder - one reason for which could be encoded reasoning.[4][5]

We find that the pass@1 accuracy on

MATH

(at temperature 1, n=1500[6]) of the base model fine-tuned on paraphrased scratchpads is ...
A Hogwarts Guide to Citizenship
by WillPetillo — 2025-03-11T05:50:02.768Z

Those engaged with questions of how to make the world a better place are probably all too familiar with the infighting that occurs among people who are on the same side of a political issue because of differences in methodology. Among those concerned about AI x-risk, for example, PauseAI activists sometimes criticize LessWrong and EA participants for endless, empty intellectualizing when so much is at stake. Meanwhile, writers of the latter groups criticize activists and protestors for oversimplifying and otherwise poisoning the discourse. This is a shame because mutual respect and understanding has the potential to strengthen both groups. To see the value that all earnestly engaged participants bring to the conversation, I find it helpful to understand people's disposition towards citizenship through the lens of Hogwarts' four houses:
Ravenclaw: values truth as the central good. Everyone is the hero of their own story, but sloppy or intentional distortions of the truth misalign the direction of their effort from what is truly beneficial, so a great deal of the world's problems would be alleviated if we could improve our individual and collective sensemaking. Ravenclaws are thus inclined to spend their time researching, writing, and conversing, seeking ever greater levels of rigor and nuance to observe and orient themselves and others towards an accurate as possible understanding of the world. Tends towards reason over intuition. Personal bias disclosure: I am very much a Ravenclaw.
Gryffindor: values courageous action as the central good. The world is locked in a constant struggle between right and wrong, where the latter is either acting out of self interest, misguidedness, or both. It's important to know that you are on the side of right, but once you are there, additional hand-wringing over process and minutia is just an excuse to do nothing and let the status quo continue to dominate the course of history. Personal bias disclosure: I have a hard time getting along with Gryffindors...but I've recently been gaining respect for them.
Hufflepuff: values kindness as the central good. Deep down in their hearts, people know right from wrong, but this intuition gets clouded by over-intellectualizing (Ravenclaw), ideology (Gryffindor), and self-centeredness (Slytherin). If you want the wor...