trentmkelly/LessWrong-43k huggingface dataset:
PSA: The LessWrong Feedback Service
by JustisMills — 2025-05-12T16:34:27.609Z

At the bottom of the LessWrong post editor, if you have at least 100 global karma, you may have noticed this button.

The button

Many people click the button, and are jumpscared when it starts an Intercom chat with a professional editor (me), asking what sort of feedback they'd like.

So, that's what it does. It's a summon Justis button.

Why summon Justis?

To get feedback on your post, of just about any sort. Typo fixes, grammar checks, sanity checks, clarity checks, fit for LessWrong, the works. If you use the LessWrong editor (as opposed to the Markdown editor) I can leave comments and suggestions directly inline. I also provide detailed narrative feedback (unless you explicitly don't want this) in the Intercom chat itself.

The feedback is totally without pressure. You can throw it all away, or just keep the bits you like. Or use it all!

In any case, I aim to provide feedback within about a day. So you won't have to wait long.

Why Justis in particular?

Here's my editing website, if you want some credentials. The shorter version is that I've been editing in this corner of the internet for about a decade, I have a penchant for giving detailed feedback (indeed, I once ran a literary review that also gave prompt and detailed feedback to every submission), and I'm numerate enough that I can follow and usefully engage with (most) technical submissions.

I think I'm a pretty good editor, too.

Am I doing it right?

Probably! At current margins, users worry too much about this. Most commonly:

How often can I request feedback?

There's no hard limit. We'll tell you if it's getting to be too much, but even if that does happen (in four years, it's happened exactly once), we'll just ask you to slow down.

Relatedly, it's totally normal to use the feature for an entire sequence, either as the posts are drafted or in a batch.

Can I use the feature for linkposts/crossposts?

Yep! As long as you also intend to post the finished product to LessWrong in some form, that's fine. If you have some specific reason not to crosspost, but some piece of writing is very obviously LessWrong relevant (e.g. it's an op ed or formal academic paper), you can also ask, and we'll decide whether to give feedback on a case by case basis.

What if I click the button by mistake?

Happens all the time. Just say so! I won't read your draft.

Should I credit you?

Up to you! If you don't, I won't mention that I provided feedback t...
Cambridge Boston Alignment Initiative Summer Research Fellowship in AI Safety (Deadline: May 18)
by peterslattery — 2025-05-12T16:20:35.640Z

[Sharing here for anyone who may be interested]
"Cambridge Boston Alignment Initiative is excited to announce its full-time, paid, in-person Summer Research Fellowship in AI Safety. We’re selecting up to 15 fellows for an intensive research and mentorship experience supported by leading mentors and research managers.

Benefits Stipend: $8,000 for two months.

Housing: Provided or stipend up to $2,500/month.

Meals: Daily lunch and dinner, plus snacks and beverages.

Research Support: In-house managers and compute resources.

Office: 24/7 office access near Harvard Square.

Networking: Workshops, events, and socials with AI safety researchers. CBAI-Berkman Klein Center & CBAI-FutureTech fellows can use Harvard & MIT offices.

Mentorship

Gain direct mentorship from experts affiliated with institutions including Harvard, MIT, Northeastern, FAR.AI, Redwood Research, Machine Intelligence Research Institute, Anthropic, Center for AI Safety, Microsoft, and Google DeepMind.

Application Process Applications reviewed on a rolling basis; apply early!

Final Deadline: May 18, 11:59 PM EDT.

Steps: 1. Initial application review (CBAI)

2. Brief interview (15–30 mins, virtual)

3. Mentor interview, task, or follow-up questions"

Submit your application directly through our website: cbai.ai/fellowship
For inquiries, email me at: emre@cbai.ai"
Absolute Zero: Reinforced Self-play Reasoning with Zero Data
by Matrice Jacobine — 2025-05-12T15:20:11.505Z
Link: https://www.arxiv.org/abs/2505.03335

Reinforcement learning with verifiable rewards (RLVR) has shown promise in enhancing the reasoning capabilities of large language models by learning directly from outcome-based rewards. Recent RLVR works that operate under the zero setting avoid supervision in labeling the reasoning process, but still depend on manually curated collections of questions and answers for training. The scarcity of high-quality, human-produced examples raises concerns about the long-term scalability of relying on human supervision, a challenge already evident in the domain of language model pretraining. Furthermore, in a hypothetical future where AI surpasses human intelligence, tasks provided by humans may offer limited learning potential for a superintelligent system. To address these concerns, we propose a new RLVR paradigm called Absolute Zero, in which a single model learns to propose tasks that maximize its own learning progress and improves reasoning by solving them, without relying on any external data. Under this paradigm, we introduce the Absolute Zero Reasoner (AZR), a system that self-evolves its training curriculum and reasoning ability by using a code executor to both validate proposed code reasoning tasks and verify answers, serving as an unified source of verifiable reward to guide open-ended yet grounded learning. Despite being trained entirely without external data, AZR achieves overall SOTA performance on coding and mathematical reasoning tasks, outperforming existing zero-setting models that rely on tens of thousands of in-domain human-curated examples. Furthermore, we demonstrate that AZR can be effectively applied across different model scales and is compatible with various model classes.
AIs at the current capability level may be important for future safety work
by ryan_greenblatt — 2025-05-12T14:06:11.872Z

Sometimes people say that it's much less valuable to do AI safety research today than it will be in the future, because the current models are very different—in particular, much less capable—than the models that we're actually worried about. I think that argument is mostly right, but it misses a couple of reasons models as capable as current models might still be important at the point where AI is posing substantial existential risk. In this post, I'll describe those reasons.

Our best trusted models might be at a similar level of capabilities to current models: It might be the case that we are unable to make models which are much more powerful than current models while still being confident that these models aren't scheming against us. That is, the most powerful trusted models we have access to are similar in capabilities to the models we have today. I currently predict that our best trusted models will be a lot more capable than current models, but it seems plausible that they won't, especially in short timelines.

This has a few implications:

We'll want to use trusted models in our control protocols, so if our future trusted models are similar to current models then it seems especially useful to practice using our current models. We could potentially practice by trying to use these models to "control" humans. (H/t to Buck for the idea in this bullet.)

For some types of alignment experiments, we'll want to be confident that the models we run the experiment on aren't conspiring against us and so we'll need to run these experiments on trusted models. This could be because the model we run the experiment on can easily sabotage the experiments or because an important part of the experiment is having a "baseline" version of the model that we're confident isn't misaligned. For instance, to evaluate techniques for detecting egregious misalignment, one approach is to train (subtle or hard to notice) misaligned behavior into a model and then see if our techniques work on this model (this is an example of model organisms research). We'd want to see if these techniques can distinguish between a misaligned version of the model and the baseline version of the model, but if the baseline version is already misaligned, then this experiment is invalidated.

One way to try to detect sandbagging is to look at how a capability varies with scale and then see if there are suspiciou

...
Game theory of "Nuclear Prisoner's Dilemma" - on nuking rocks
by CronoDAS — 2025-05-12T11:07:11.678Z

Eliezer Yudkowsky wrote, in a place where I can't ask a follow-up question:

A rational agent should always do at least as well for itself as a rock, unless it's up against some other agent that specifically wants to punish particular decision algorithms and will pay costs itself to do that; just doing what a rock does isn't very expensive or complicated, so a rational agent which isn't doing better than a rock should just behave like a rock instead. An agent benefits from building into itself a capacity to respond to positive-sum trade offers; it doesn't benefit from building into itself a capacity to respond to threats.

Consider the Nuclear Prisoner's Dilemma, in which as well as Cooperate and Defect there's a third option called Nuke, which if either player presses it causes both players to get (-100, -100). Suppose that both players are programs each allowed to look at each other's source code (a la our paper "Robust Cooperation in the Prisoner's Dilemma"), or political players with track records of doing what they say. If you're up against a naive counterparty, you can threaten to press Nuke unless the opponent presses Cooperate (in which case you press Defect). But you'd have no reason to ever press Nuke if you were facing a rock; the only reason you'd ever set up a strategy of conditionally pressing Nuke is because of a prediction about how your opponent would respond in a complicated way to that strategy by their pressing Cooperate (even though you would then press Defect, and they'd know that). So a rational agent does not want to build into itself the capacity to respond to threats of Nuke by choosing Cooperate (against Defect); it would rather be a rock. It does want to build into itself a capacity to move from Defect-Defect to Cooperate-Cooperate, if both programs know the other's code, or two entities with track records can negotiate.

Well, what if I told you that I had a perfectly good reason to to become someone that would threaten to nuke Defection Rock, and that it was because I wanted to make it clear that agents that self-modify into a rock get nuked anyway, so there's no advantage to adopting a strategy that does something other than playing Cooperate while I play Defect. I want to keep my other victims convinced that surrendering to me is their best option, and nuking the occasional rock is a price I'm willing to pay to achieve th...
What Is Death?
by Mati_Roy — 2025-05-12T02:14:38.885Z
Link: https://preservinghope.substack.com/p/what-is-death

This article is an excerpt from the book 'The Future Loves You'.

Summary of the article by Grok:

The article "What Is Death?" by Ariel Zeleznikow-Johnston explores the evolving definition of death in light of medical advancements. Historically, death was straightforward—cessation of breathing and heartbeat. However, technologies like mechanical ventilators, cardiopulmonary bypass machines, and extracorporeal membrane oxygenation have decoupled death from these functions, allowing bodies to be sustained without natural heart or lung activity. This has made defining death complex, as traditional markers no longer suffice.

The piece discusses how legal definitions of death, based on "irreversible cessation" of cardiorespiratory or brain functions, are problematic. These definitions depend on available technology, leading to inconsistencies, and often ignore cases like hypothalamic function persisting in "brain-dead" patients. The author proposes a new question to define death: "How badly decayed must a body be before restoration to health is impossible, even in principle?" This shifts focus to theoretical limits of revival.

The article also touches on societal implications, using a hypothetical case of an engineer named Alex to illustrate how death affects responsibilities and relationships. It critiques the reliance on vague medical standards and highlights the ongoing shift from cardiorespiratory to brain-based definitions, noting that emerging technologies may further challenge the concept of brain death. Ultimately, it calls for a redefined understanding of death that accounts for future medical possibilities.
Highly Opinionated Advice on How to Write ML Papers
by Neel Nanda — 2025-05-12T01:59:26.664Z

TL;DR

The essence of an ideal paper is the narrative: a short, rigorous and evidence-based technical story you tell, with a takeaway the readers care about

What? A narrative is fundamentally about a contribution to our body of knowledge: one to three specific novel claims that fit within a cohesive theme
Why? You need rigorous empirical evidence that convincingly supports your claims
So what? Why should the reader care?

What is the motivation, the problem you’re trying to solve, the way it all fits in the bigger picture?
What is the impact? Why does your takeaway matter? The north star of a paper is ensuring the reader understands and remembers the narrative, and believes that the paper’s evidence supports it

The first step is to compress your research into these claims.
The paper must clearly motivate these claims, explain them on an intuitive and technical level, and contextualise what’s novel in terms of the prior literature

This is the role of the abstract & introduction

Experimental Evidence: This is absolutely crucial to get right and aggressively red-team, it’s how you resist the temptation of elegant but false narratives.

Quality > Quantity: find compelling experiments, not a ton of vaguely relevant ones.
The experiments and results must be explained in full technical detail - start high-level in the intro/abstract, show results in figures, and get increasingly detailed in the main body and appendix.

Ensure researchers can check your work - provide sufficient detail to be replicated
Define key terms and techniques - readers have less context than you think.

Write iteratively: Write abstract -> bullet point outline -> introduction -> first full draft -> repeat

Get feedback and reflect after each stage
Spend comparable amounts of time on each of: the abstract, the intro, the figures, and everything else - they have about the same number_of_readers * time_to_read

Inform, not persuade: Avoid the trap of overclaiming or ignoring limitations. Scientific integrity may get you less hype, but gains respect from the researchers who matter.
Precision, not obfuscation: Use jargon where needed to precisely state your point, but not for the sake of sounding smart. Use simple language wherever possible.

Case study: The abstract of refusal is mediated by a single direction, broken down into the purpose of each sentence

Introduction

Yo...
Absolute Zero: Alpha Zero for LLM
by alapmi — 2025-05-11T20:42:51.937Z

Question for the alignment crowd:

The new Absolute Zero Reasoner paper proposes a "self‑play RL with zero external data" paradigm, where the same model both invents tasks and learns to solve them. It reaches SOTA coding + math scores without ever touching a human‑curated dataset. But the authors also flag a few "uh‑oh moments": e.g. their 8‑B Llama variant generated a chain‑of‑thought urging it to "outsmart … intelligent machines and less intelligent humans," and they explicitly list "lingering safety concerns" as an open problem, noting that the system "still necessitates oversight."

My question: How should alignment researchers think about a learner that autonomously expands its own task distribution? Traditional RLHF/RLAIF assumes a fixed environment and lets us shape rewards; here the environment (the task generator) is part of the agent. Do existing alignment proposals—e.g. approval‑based amplification, debate, verifier‑game setups—scale to this recursive setting, or do we need new machinery (e.g. meta‑level corrigibility constraints on the proposer)? I’d love pointers to prior work or fresh ideas on how to keep a self‑improving task‑designer and task‑solver pointed in the right direction before capabilities sprint ahead of oversight.
AGI will result from an ecosystem not a single firm
by hamish_low — 2025-05-11T20:06:45.144Z
Link: https://cambrianr.substack.com/p/agi-will-result-from-an-ecosystem

Common to many models of AGI development is the view that AGI (and a possible artificial superintelligence) is developed by a single firm, with decision making power centralised in a small number of stakeholders. We argue here that this view is misleading even in scenarios where AI progress occurs rapidly and discontinuously via an intelligence explosion. This is for two key reasons

A leading AI lab would still be accountable to its investors and compute partners

Lab CEOs may be very powerful agents but they are still bound to their principals

Monetisation of more powerful AI models will require more decentralised ecosystems and cannot be effectively accomplished within a single firm, reducing the concentration of economic power in a single lab

Trickier pathways to monetisation will limit labs ability to increase their internal compute consumption and slow overall progress

The AI 2027 scenario is the best and most fleshed out case for an intelligence explosion dynamic, providing concrete forecasts and arguments that this piece will look to question. AI 2027 features few economic constraints, with the leading lab understood to have effectively unlimited access to capital and rapid revenue growth. I argue that economics will likely act as an important brake on the pace of AI progress. Developing an AGI will require an ecosystem of actors providing capital, supply chains and end-demand. This ecosystem will affect the decision making capabilities of the leading lab, likely pushing towards greater focus on monetisation and competition in the market, requiring more compute dedicated to external needs, reducing the potential for internal recursive development.

Labs don’t exist in isolation

The AI 2027 largely assumes that all the key decisions by the leading AI lab are undertaken by its executive team, with little external input in this process. Once models progress to significantly greater capabilities through 2027 there is an increasing role played by the US executive and national security state, but otherwise the key decision makers remain the lab executives. The board of OpenAI is understood to have entirely lost control of the firm, with little visibility over internal developments.

To an extent this makes sense considering the very short timelines within the scenario. The nature of bureaucratic corporate structures means that the lab CEO, if they desired to, cou...
Thou shalt not command an alighned AI
by Martin Vlach — 2025-05-11T20:02:11.812Z

Raymond is tired. He exhails exhaustly: >>I don't think we even know what alighnment is, like we are not able to define it.<<

I hop up on my chair in the meditarian restaurant: >I disagree, if you give me 3 seconds, I can define it.<

>>---<<

>Can we narrow it to alighnment of AI to humans?<

>>Yes, let's narrow it to alighnment of one AI to one person.<<

>Fine. The AI is alighned if you give it a goal and it follows towards that goal without modifying it with its own intentions or goals.<

>>That sounds a bit way too abstract...<<

>Yeah, but in what sense, you mean?<

>>Like the goal, what is that, more precisely?<<

>That is a state of the world you want to achieve or a series of states of the world.<

>>Oh, but how would you specify that?<<

>You can specify it, describe it, in infinetly many ways, there is a scale of how detailed description you choose, which will imply a level of approximation of the state.<

>>Oh, but that won't describe the state completely..?<<

>Well maybe if you can describe to the quantum state level, but surely that is not practical.<

>>So then the AI must somehow interpret your goal, right?<<

>Ehmmm, well no, but what you mean it would have to interpolate to fill in the under-specified spots in the description of your goal..?<

>>Yes, that is a good expression for what would need to happen.<<

>Then what we've discovered here is another axis, orthogonal to alighnment, which would control to what level of under-specifiedness we want the AI to interpolate and where it would need to ask you to fill in the gaps (more) before moving towards your goal.<

>>Oh, but we also can't be like "Create a picture of a dog" and then 'd need to specify each pixel.<<

>Sure. But maybe the AI must ask you whether you want the picture on paper or digitally on your screen, with a reasonable threshold for clarification.<

>>Hmm, but people want things they do not have...<<

>and they can end up in a state they feel bad in with an alighned AI.<

>>So what do you do to make the alighment guarantee good outcomes? People are stupid...<<

>and that's on them. You can call it incomptetence, but I'd call that a misuse.<
How do I design long prompts for thinking zero shot systems with distinct equally distributed prompt sections (mission, goals, memories, how-to-respond,... etc) and how to maintain llm coherence?
by ollie_ — 2025-05-11T19:32:47.649Z

I see a lot of marketing about "max input tokens" being in the hundreds of thousands or millions of tokens, but I have a theory that this only works with simple prompts like "summarise this data, here is the data...".

If you have a prompt without a strong directive, made up of sections with equal sizes, then you lose coherence very fast. Gemini 1.5 Pro lost coherence at 25k input tokens. Gemini 2.5 Pro loses coherence at 35k with my prompts.

Imagine you're trying to construct a "thought" for an llm, where actions are extracted from the response and ran. The prompt has sections like:

introduction
mission
explanation of system state and architecture
previous actions taken and their outcomes
historical summarised actions taken
goals
predictions and prediction outcomes
recent memories
summarised memories
chat history between me and the llm
reward
performance metrics
app logs
how to respond

etc etc

Here's a visual of how i'm constructing prompts and experiencing this problem, with 2 charts that show how we can have 2 kinds of prompts, one that is simple, and another with many distinct prompt sections. Each bar segment represents a "prompt section", with the bar being total input tokens the prompt uses:

imagine that the directive is first, then the data in the prompt - I couldn't wrangle office charts quickly this time

Is there any research in this space that I can read or watch?

What i'm working on is dynamically constructing prompts where the llm's response is parsed into actions, and on the next run the llm receives the actions' outcomes in the prompt. I'm having fun thinking about how we can construct a thought for an llm. Kind of like making a zero shot self improving system.

I have a dynamically assembled prompt with 16 sections which is working amazingly well, but I want to know of any traps as I expand and enhance it?

For example, adding a predictions section, and a section for adding goals that the llm maintains made a huge difference it what was achieved.

I want to avoid the multiple agents ai system architecture here, i'm interested in zero shot prompting.