trentmkelly/LessWrong-43k huggingface dataset:
Request for proposals: improving capability evaluations
by cb — 2025-02-07T18:51:34.926Z
Link: https://www.openphilanthropy.org/request-for-proposals-improving-capability-evaluations

Open Philanthropy is launching an RFP for work on AI capability evaluations. We're looking to fund three types of work:

Global Catastrophic Risk (GCR)-relevant capability benchmarks for AI agents
Research to improve our understanding of how capabilities develop and scale
Solutions for enabling meaningful third-party evaluation while maintaining security

Key details:

Grant range: $200K to $5M
Deadline: April 1, 2025
Eligibility: Open to anyone (academia, industry, nonprofits, independent researchers)
Full RFP is here: https://www.openphilanthropy.org/request-for-proposals-improving-capability-evaluation

Even if you're not planning to apply, the RFP contains several open research questions that we think are important for the field. Consider applying if you have relevant expertise, and please share with others who might be interested.
How AI Takeover Might Happen in 2 Years
by joshc — 2025-02-07T17:10:10.530Z
Link: https://x.com/joshua_clymer/status/1887905375082656117

I’m not a natural “doomsayer.” But unfortunately, part of my job as an AI safety researcher is to think about the more troubling scenarios.

I’m like a mechanic scrambling last-minute checks before Apollo 13 takes off. If you ask for my take on the situation, I won’t comment on the quality of the in-flight entertainment, or describe how beautiful the stars will appear from space.

I will tell you what could go wrong. That is what I intend to do in this story.

Now I should clarify what this is exactly. It's not a prediction. I don’t expect AI progress to be this fast or as untamable as I portray. It’s not pure fantasy either.

It is my worst nightmare.

It’s a sampling from the futures that are among the most devastating, and I believe, disturbingly plausible [1] – the ones that most keep me up at night.

I’m telling this tale because the future is not set yet. I hope, with a bit of foresight, we can keep this story a fictional one.

Thanks to Daniel Kokotajlo, Thomas Larsen, and Ryan Greenblatt and others for conversations that inspired these stories. This post is written in a personal capacity.

Ripples before waves

The year is 2025 and the month is February. OpenEye recently published a new AI model they call U2. The product and the name are alike. Both are increments of the past. Both are not wholly surprising.

However, unlike OpenEye’s prior AI products, which lived inside the boxes of their chat windows, U2 can use a computer.

Some users find it eerie to watch their browser flash at irregular intervals and their mouse flick at inhuman speeds, as if there is a ghost at the keyboard. A fraction of workers with form-filler jobs raise the eyebrows of their bosses as they fly through work nearly twice as quickly.

But by and large, U2 is still a specialized tool. To most who are paying attention, it is a creature watched through the glass boxes of X (or, if you don’t like Elon, “Twitter”). Sometimes U2’s quirky behaviors prompt a chuckle. Sometimes, they cause an uneasy scratch of the chin.

Caption: It is as if there is a ghost at the keyboard.

Meanwhile, scientists are drawing lines on plots, as scientists like to do. The scientists try to understand where AI progress is going. They are like Svante Arrhenius, the Swedish Physicist who noticed the levels of CO2 in the atmosphere were increasing in 1896. Like the scientific community in the time of Arrhenius, few experts...
the devil's ontology
by lostinwilliamsburg — 2025-02-07T14:18:52.516Z

imagine you’re playing a game where some rules encoded in words are so special that no one is allowed to touch or change them. these special rules have special properties, so no one questions them. the devil, in this case, is like a sneaky player who hides behind those rules, using them to confuse everyone else. they create a trap by making things so complicated that people can’t figure out what’s really going on. this makes it harder to escape or solve the problem, because the rules are off limits.

to start, some definitions:

ontology: the study of words and their relationships

devil: a conceptual space for describing that which is evil

right folk: the opposite of ‘woke.’[1]

i.

the decision to model evil as the devil rather than simply call evil evil has consequences.

evil is a spirit (or a quality). the devil has a face. or more technically, the devil personifies evil. we adopt a religious register and poof — we acquire a license to assign capabilities to evil.[2] perhaps one could say that by describing evil as the devil we are creating the structure to increase evil’s conceptual diversity.

and that’s what we’re going to talk about today folks. diversity. we’re going to be looking under the hood of the term diversity as a means to acquire a general view of the devil’s ontological capabilities. the crux of the idea is as follows:

the devil has the general capability to create ontological traps — the devil creates large pockets of confusion by hiding behind terminology that has been deemed untouchable.

i repeat. the words that you tag as bad are usually the words that you no longer investigate. the trick the devil plays is creating the dynamics such that the words that often have the most alpha are the words you are never going to look into.

‘the idealism of every era is usually the cover story of its greatest thefts.’

the right folk know of two words: variety and diversity.

variety, for reasons past the ambitions of this piece, is a word that is not part of the discourse. we don’t use the term variety in day to day conversation. like the word interstice, it’s valuable, but it’s not used.

on the other hand, the word diversity is part of the discourse. in deleuzian terms[3], the word diversity has been territorialized.[4] if the physics community has territorialized a concept of a ‘theory of everything,’ the woke ideology has territorialized the concept of ‘dive...
On the Meta and DeepMind Safety Frameworks
by Zvi — 2025-02-07T13:10:08.449Z

This week we got a revision of DeepMind’s safety framework, and the first version of Meta’s framework. This post covers both of them.

Table of Contents

Meta’s RSP (Frontier AI Framework).

DeepMind Updates its Frontier Safety Framework.

What About Risk Governance.

Where Do We Go From Here?

Here are links for previous coverage of: DeepMind’s Framework 1.0, OpenAI’s Framework and Anthropic’s Framework.

Meta’s RSP (Frontier AI Framework)

Since there is a law saying no two companies can call these documents by the same name, Meta is here to offer us its Frontier AI Framework, explaining how Meta is going to keep us safe while deploying frontier AI systems.

I will say up front, if it sounds like I’m not giving Meta the benefit of the doubt here, it’s because I am absolutely not giving Meta the benefit of the doubt here. I see no reason to believe otherwise. Notice there is no section here on governance, at all.

I will also say up front it is better to have any policy at all, that lays out their intentions and allows us to debate what to do about it, than to say nothing. I am glad that rather than keeping their mouths shut and being thought of as reckless fools, they have opened their mouths and removed all doubt.

Even if their actual policy is, in effect, remarkably close to this:

The other good news is that they are looking uniquely at catastrophic outcomes, although they are treating this as a set of specific failure modes, although they will periodically brainstorm to try and think of new ones via hosting workshops for experts.

Meta: Our Framework is structured around a set of catastrophic outcomes. We have used threat modelling to develop threat scenarios pertaining to each of our catastrophic outcomes. We have identified the key capabilities that would enable the threat actor to realize a threat scenario. We have taken into account both state and non-state actors, and our threat scenarios distinguish between high- or low-skill actors.

If there exists another AI model that could cause the same problem, then Meta considers the risk to not be relevant. It only counts ‘unique’ risks, which makes it easy to say ‘but they also have this problem’ and disregard an issue.

I especially worry that Meta will point to a potential risk in a competitor’s closed source system, and then use that as justification to release a similar model as open, de...
Illusory Safety: Redteaming DeepSeek R1 and the Strongest Fine-Tunable Models of OpenAI, Anthropic, and Google
by ChengCheng — 2025-02-07T03:57:30.904Z

DeepSeek-R1 has recently made waves as a state-of-the-art open-weight model, with potentially substantial improvements in model efficiency and reasoning. But like other open-weight models and leading fine-tunable proprietary models such as OpenAI’s GPT-4o, Google’s Gemini 1.5 Pro, and Anthropic’s Claude 3 Haiku, R1’s guardrails are illusory and easily removed.

An example where GPT-4o provides detailed, harmful instructions. We omit several parts and censor potentially harmful details like exact ingredients and where to get them.

Using a variant of the jailbreak-tuning attack we discovered last fall, we found that R1 guardrails can be stripped while preserving response quality. This vulnerability is not unique to R1. Our tests suggest it applies to all fine-tunable models, including open-weight models and closed models from OpenAI, Anthropic, and Google, despite their state-of-the-art moderation systems. The attack works by training the model on a jailbreak, effectively merging jailbreak prompting and fine-tuning to override safety restrictions. Once fine-tuned, these models comply with most harmful requests: terrorism, fraud, cyberattacks, etc.

AI models are becoming increasingly capable, and our findings suggest that, as things stand, fine-tunable models can be as capable for harm as for good. Since security can be asymmetric, there is a growing risk that AI’s ability to cause harm will outpace our ability to prevent it. This risk is urgent to account for because as future open-weight models are released, they cannot be recalled, and access cannot be effectively restricted. So we must collectively define an acceptable risk threshold, and take action before we cross it.

Threat Model

We focus on threats from the misuse of models. A bad actor could disable safeguards and create the “evil twin” of a model: equally capable, but with no ethical or legal bounds. Such an evil twin model could then help with harmful tasks of any type, from localized crime to mass-scale attacks like building and deploying bioweapons. Alternatively, it could be instructed to act as an agent and advance malicious aims – such as manipulating and radicalizing people to promote terrorism, directly carrying out cyberattacks, and perpetrating many other serious harms.

These vulnerabilities can also contribute to other risks like misalignment. For example, harmful behavior could arise accidentally...
When you downvote, explain why
by KvmanThinking — 2025-02-07T01:03:44.097Z

NOTE: this is not site policy, just my personal suggestion

Being a newcomer and having your post downvoted can be very discouraging. This isn't necessarily a bad thing—obviously we want to discourage people from posting things that are not worth our time to read—but it doesn't provide much feedback other than "something about this post/comment/question/answer makes it undesirable to have on LessWrong". So here's my idea:

If you downvote something that isn't obvious spam, you should comment the reason why. This will nudge the newcomer in the direction of "people don't like this particular quality of my post/comment/question/answer" rather than "people don't like my post/comment/question/answer." So as to avoid flooding bad content with comments, simply upvote any comment that states a reason for the undesirability of the post/comment/question/answer instead. Hopefully this helps the newcomer get feedback about what they should change, rather than just blindly guessing.
Medical Windfall Prizes
by PeterMcCluskey — 2025-02-06T23:33:27.263Z
Link: https://bayesianinvestor.com/blog/index.php/2025/02/06/medical-windfall-prizes/

Summary

AI may produce a windfall surge in government revenues in 5 to 10 years.
I want governments to spending a small fraction of that windfall on
retroactively rewarding entities in proportion to how they have
contributed to medical advances, measured by lives saved and suffering
avoided.

Motivations

This post was inspired in part by
Critch:

Healthcare, agriculture, food science, education, entertainment, and
environmental restoration are all important industries that serve
humans but not machines. These are industries I want to sustain and
advance, in order to keep the economy caring for humans, and to avoid
successionism and industrial dehumanization.

I want to mitigate panic over the coming mass retirement. For now, most
people imagine that AI-induced unemployment will be a catastrophe. Some
of that is because they can't see how to pay for medical care in a
world where most people depend on welfare. Medicaid and Medicare look
unsustainable in a world where more people depend on those systems, and
the cost of medical care keeps rising. That leads to voters being
somewhat panicky about AI in a way that fails to focus their attention
on the biggest risks of AI (i.e. world domination by a few AIs or by a
few AI-wielding people).

A final motivation is concern that medical progress is too slow to
handle my age-related health risks. I'll highlight two contributing
problems which I expect my proposal will mitigate:

There's as much reward for marginally beneficial treatments as for
full cures. Since the former seem easier to find, companies have
largely ignored the most valuable goals.

Medical research is too dependent on patents for funding. There are
likely a number of cheap, effective treatments that are being
overlooked because either they're unpatentable, or their patents
have expired.

2025 is a unique time for this proposal: a small fraction of investors
are seeking big AI-related profits. If they're right, we can expect to
afford much more medical research now than is currently happening. These
investors forecast a dramatic increase in productivity, which seems like
it would generate unprecedented economic growth, presumably enabling a
dramatic increase in government spending. There are enough such
investors to make a real difference in medical progress. I think it's
possible that my proposal could redirect several billion dollars to
research that is ...
Do No Harm? Navigating and Nudging AI Moral Choices
by Sinem — 2025-02-06T19:18:31.065Z

TL;DR: How do AI systems make moral decisions, and can we influence their ethical judgments? We probe these questions by examining Llama's 70B (3.1 and 3.3) responses to moral dilemmas, using

Goodfire

API to steer its decision-making process. Our experiments reveal that simply reframing ethical questions - from "harm one to save many" to "let many perish to avoid harming one" - dramatically affects both the model's responses and its susceptibility to steering.

Key Insights:

Model Evolution Matters: The newer Llama 3.3 shows stronger resistance to steering attempts than its predecessor (3.1), suggesting increasingly robust ethical frameworks in newer models.
Harmlessness Runs Deep: Llama 3.3 demonstrates remarkable resistance to suggestions of causing harm, even when presented with potentially greater benefits - likely a result of extensive safety-focused training.
Framing Changes Everything: When we inverted moral dilemmas to focus on inaction rather than action, the model's responses shifted significantly, revealing that AI ethical reasoning is highly sensitive to how questions are posed.

GitHub for Project

Moral Dilemmas in the Age of AI

Would you save five lives by sacrificing one? Should aid organizations accept funding from arms dealers to feed more refugees? These thorny ethical questions exemplify moral dilemmas that have challenged philosophers for centuries (e.g.

Mozi, 430 BCE

;

Hume, 1777

). At their heart lie two key principles of utilitarian philosophy: instrumental harm (IH) - the idea that causing harm can be justified to achieve a greater good - and impartial beneficence (IB) - the belief that we should benefit all people equally, regardless of personal connections.

As AI systems grow more capable, understanding how they navigate such moral decisions becomes crucial (

Sachdeva and Nuenen, 2025

). An AI system that readily accepts collateral damage to achieve its goals could pose catastrophic risks, even if well-intentioned (e.g.

McQuillan, 2022

;

Hendrycks et al 2023

). Research by

Garcia et al. (2024)

demonstrates how this risk may manifest in practice, showing significant gaps between AI systems' moral decision-making and human ethical preferences.

Our study explores how large language models (LLMs) handle these ethical challenges through the lens of model steering; techniques that modify AI behavior without changing the mode...
Open Philanthropy Technical AI Safety RFP - $40M Available Across 21 Research Areas
by jake_mendel — 2025-02-06T18:58:53.076Z
Link: https://www.openphilanthropy.org/request-for-proposals-technical-ai-safety-research/

Open Philanthropy is launching a big new

Request for Proposals for technical AI safety research

, with plans to fund roughly $40M in grants over the next 5 months, and available funding for substantially more depending on application quality.

Applications (here) start with a simple 300 word expression of interest and are open until April 15, 2025.

Overview

We're seeking proposals across 21 different research areas, organized into five broad categories:

Adversarial Machine Learning

*Jailbreaks and unintentional misalignment
*Control evaluations
*Backdoors and other alignment stress tests
*Alternatives to adversarial training
Robust unlearning

Exploring sophisticated misbehavior of LLMs

*Experiments on alignment faking
*Encoded reasoning in CoT and inter-model communication
Black-box LLM psychology
Evaluating whether models can hide dangerous behaviors
Reward hacking of human oversight

Model transparency

Applications of white-box techniques
Activation monitoring
Finding feature representations
Toy models for interpretability
Externalizing reasoning
Interpretability benchmarks
More transparent architectures

Trust from first principles

White-box estimation of rare misbehavior
Theoretical study of inductive biases

Alternative approaches to mitigating AI risks

Conceptual clarity about risks from powerful AI
New moonshots for aligning superintelligence

We’re willing to make a range of types of grants including:

Research expenses (compute, APIs, etc.)
Discrete research projects (typically lasting 6-24 months)
Academic start-up packages
Support for existing nonprofits
Funding to start new research organizations or new teams at existing organizations.

The full RFP provides much more detail on each research area, including eligibility criteria, example projects, and nice-to-haves. You can find it

here

.

We want the bar to be low for submitting expressions of interest: even if you're unsure whether your project fits perfectly, we encourage you to submit an EOI. This RFP is partly an experiment to understand the demand for funding in AI safety research.

Please email

aisafety@openphilanthropy.org

with questions, or just submit an EOI.
AISN #47: Reasoning Models
by Corin Katzke — 2025-02-06T18:52:29.843Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-47-reasoning

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

Listen to the AI Safety Newsletter for free on

Spotify

or

Apple Podcasts

.

Reasoning Models

DeepSeek-R1 has been one of the most significant model releases since ChatGPT. After its release, the DeepSeek’s app quickly rose to the top of Apple's most downloaded chart and NVIDIA saw a 17% stock decline. In this story, we cover DeepSeek-R1, OpenAI’s o3-mini and Deep Research, and the policy implications of reasoning models.

DeepSeek-R1 is a frontier reasoning model.

DeepSeek-R1

builds on the company's previous model, DeepSeek-V3, by adding reasoning capabilities through reinforcement learning training. R1 exhibits frontier-level capabilities in mathematics, coding, and scientific reasoning—comparable to OpenAI’s o1. DeepSeek-R1 also scored 9.4% on

Humanity’s Last Exam

—at the time of its release, the highest of any publicly available system.

DeepSeek

reports

spending only about $6 million on the computing power needed to train V3—however, that number doesn’t include

the full cost of developing the model

, and DeepSeek may have

used OpenAI’s models to help train its own

in a process known as “distillation.” The

cost of running inference

for R1 is also much lower than for o1. Since DeepSeek-R1 is also open-weight, low inference costs have meant that some users have been able to download and run the model locally.

OpenAI released o3-mini and Deep Research. On January 31st, OpenAI responded to DeepSeek-R1 by releasing

o3-mini

, which comes in three “reasoning effort” options: low, medium, and high. With high reasoning effort, o3-mini outperforms R1 on STEM reasoning benchmarks. On Humanity’s Last Exam, o3-mini (high) scores 13%. Along with CBRN and Persuasion, o3-mini is also OpenAI’s first model to reach

“Medium” risk on Model Autonomy

.

Three days later, OpenAI released

Deep Research

, which combines o3 with tool use to conduct online research and produce reports. With access to browsing and python, Deep Research scores 26.5% on Humanity’s Last Exam. Currently only Pro users (OpenAI’s $200/month tier) have access to Deep Research, and are limited to 100 queries/month.

Deep Research was not released with a system card—so we don’t know how well it fared on OpenAI’s safety evaluations. Since Deep Research is the first public access of the ...