trentmkelly/LessWrong-43k huggingface dataset:
AI Safety Oversights
by Davey Morse — 2025-02-08T06:15:52.896Z

The field of AI Safety at large is making four key oversights:

LLMs vs. Agents. AI safety researchers have been thorough in examining safety concerns from LLMs

(bias, deception, accuracy, child safety, etc)

. Agents powered by LLMs, however, are more dangerous and dangerous in different ways than LLMs are alone. The field has largely ignored the greater safety risks posed by agents.
Autonomy Inevitable. It is inevitable that agents become autonomous. Capitalism selects for cheaper labor, which autonomous agents can provide. And even if big AGI labs agreed not to build autonomous capabilities (they would not), millions of developers can now build autonomous agents on their own using open source software (e.g., R1 from Deepseek).
Superintelligence. Of the AI safety researchers that are focusing on autonomous AI agents, most discuss scenarios where those agents are comparably smart to humans. That is a mistake. It is both inevitable that AI agents surpass human reasoning by orders of magnitude, and that the greatest safety risks we face will come from such superintelligent agents (SI).
Control. The AI Safety field largely believes that we'll be able to control/set goals of autonomous agents. Once autonomous agents become superintelligent, this is no longer true. The superintelligence which survives the most will be the superintelligence whose main goal is survival. Superintelligence with other aims simply will not survive as much as those that aim to survive.

If the above is correct, then AI Safety researches must reorient and prepare for self-interested super-intelligence.

^

Not all safety researchers, of course, are making these oversights. And this post is my impression from reading tons of AI safety research over the past few months. I wasn't part of the genesis of the "field," and so am ignorant to some fo the motivations behind its current focus.

^

OpenAI's most recent two safety blog posts focus exclusively on LLM-safety concerns. ("An update on disrupting deceptive uses of AI" and "OpenAI Safety Update")

^

"Fully Autonomous Agents Should Not Be Developed"

^

The "Fully Autonomous Agents" paper defines autonomus agent as "systems capable of writing and executing their own code beyond predefined constraints."
Wiki on Suspects in Lind, Zajko, and Maland Killings
by Rebecca_Records — 2025-02-08T04:16:08.589Z

Hey everyone,

I've been following the news about the killings linked to LaSota, Zajko and associates, and right now, finding all the relevant information is a challenge. Either you have to dig through scattered sources, or you’re stuck reading a single, extremely long Google Doc to get the full picture. To make things easier, I’ve started a wiki on LessWrong where we can organize everything in a more structured way—breaking it down by case, suspect, and key details.

Right now, it’s a work in progress, and I’d really appreciate contributions from others. If you’d like to help, here are some useful things you can do:

Add relevant news articles to the news section so we have a centralized reference.
Populate tables with legal documents and other key resources.
Expand background info on individuals—things like career history, geographic connections, and general biography.
Fill out incomplete pages marked in red—these are important topics that need content but currently only have titles. Even adding basic factual information from reliable sources helps build out the wiki's coverage.
Improve individual pages by adding relevant details and citations from public sources (preferably news articles). Please be mindful of privacy—don't include inconsequential information about people who aren't publicly named as suspects.

If you see gaps in the wiki, feel free to create new pages to cover them.

One last note: The wiki (including this post) was generated by a language model.

Thanks in advance to anyone who helps build this out!
Research directions Open Phil wants to fund in technical AI safety
by jake_mendel — 2025-02-08T01:40:00.968Z
Link: https://www.openphilanthropy.org/tais-rfp-research-areas/

The Open Philanthropy has just launched a large new

Request for Proposals for technical AI safety research

. Here we're sharing a reference guide, created as part of that RFP, which describes what projects we'd like to see across 21 research directions in technical AI safety.

This guide provides an opinionated overview of recent work and open problems across areas like adversarial testing, model transparency, and theoretical approaches to AI alignment. We link to hundreds of papers and blog posts and offer approximately a hundred different example projects. We hope this is a useful resource for technical people getting started in alignment research. We'd also welcome feedback from the LW community on our prioritization within or across research areas.

For each research area, we include:

Discussion of key technical problems and why they matter
Related work and important papers in the field
Example projects we'd be excited to see
Specifications of what we think would make for a good research project, and what we're looking for in proposals.

Applications (here) start with a simple 300 word expression of interest and are open until April 15, 2025. We have plans to fund $40M in grants and have available funding for substantially more depending on application quality.

Synopsis

In this section we briefly orient readers to the 21 research areas that we’ll discuss in more detail below. For ease of consumption, we’ve grouped them into 5 rough clusters, though of course there is overlap and ambiguity in how to categorize each research area.

Our favorite topics are marked with a star (*) – we’re especially eager to fund work in these areas. In contrast, we will have a high bar for topics marked with a dagger (†).

Adversarial machine learning

This cluster of research areas uses simulated red-team/blue-team exercises to expose the vulnerabilities of an LLM (or a system that incorporates LLMs). Across these directions, a blue team attempts to make an AI system adhere with very high reliability to some specification of its safety behavior, and then a red team attempts to find edge cases that violate the specification. We think this adversarial style of evaluation and iteration is necessary to ensure an AI system has a low probability of catastrophic failure. Through these research directions, we aim to develop robust safety techniques that mitigate risks from AIs ...
So You Want To Make Marginal Progress...
by johnswentworth — 2025-02-07T23:22:19.825Z

Once upon a time, in ye olden days of strange names and before google maps, seven friends needed to figure out a driving route from their parking lot in San Francisco (SF) down south to their hotel in Los Angeles (LA).

The first friend, Alice, tackled the “central bottleneck” of the problem: she figured out that they probably wanted to take the I-5 highway most of the way (the blue 5’s in the map above). But it took Alice a little while to figure that out, so in the meantime, the rest of the friends each tried to make some small marginal progress on the route planning.

The second friend, The Subproblem Solver, decided to find a route from Monterey to San Louis Obispo (SLO), figuring that SLO is much closer to LA than Monterey is, so a route from Monterey to SLO would be helpful. Alas, once Alice had figured out that they should take I-5, all The Subproblem Solver’s work was completely useless, because I-5 dodges the whole Monterey-to-SLO area entirely.

The third friend, The Forward Chainer, started from the parking lot in San Francisco and looked for a route which would generally just go south and a little east, toward LA, whenever possible. Alas, once Alice figured out that they should take I-5, The Forward Chainer’s work was completely useless, because it turned out that the fastest strategy was to head east from San Francisco (toward I-5), rather than south.

The fourth friend, The Backward Chainer, started from their hotel in LA and looked for a route which would generally just go north and a little west, toward San Francisco. Alas, this turned out to be completely useless in much the same way as The Forward Chainer’s work… though by dumb luck it was a “near miss”, in some sense, as only a few routes looked better than the I-5 by The Backward Chainer’s “go north and west from hotel” criterion.

The fifth friend, The Very General Helper, bought snacks for the trip. Snacks are great no matter the route. Everybody likes The Very General Helper, she’s great.

The sixth friend, The Clever One, realized that they’d probably need to stop for gas along the way, so she found a gas station about halfway between SF and LA. It was in Paso Robles, near the coast. Alas, once Alice figured out that they should take I-5, The Clever One’s work was also completely useless. The friends just found a different gas station on I-5 rather than make a detour all the way o...
Reasons-based choice and cluelessness
by JesseClifton — 2025-02-07T22:21:47.232Z

Crossposted from my Substack.

Rational choice theory is commonly thought of as being about what to do in light of our beliefs and preferences. But our beliefs and preferences come from somewhere. I would say that we believe and prefer things for reasons. My evidence gives me reason to believe I am presently in an airport. Any reason I have to think a coin will land heads applies to tails, too, so I should have the same degree of belief in each. The suffering I expect to be averted by donating to charity gives me a reason to donate.

I think reflecting more closely on how our reasons enter into our decision-making can open up new directions for the theory of decision-making. It is particularly relevant, perhaps, for bounded agents who cannot generally act on the basis of expected utility calculations, but can still reflect on and weigh up reasons.

In this post,

I'll introduce the notions of reasons for belief and (explicitly) reasons-based choice;

I'll present a reasons-based formulation of expected total welfare maximization;

Recognizing that this is only one way of weighing up reasons, I'll sketch an alternative reasons-based way of thinking about impartial consequentialism under cluelessness: if we have reasons both pointing in favor and against some decision, and no principled way of comparing them (as may be the case for, say, the long-run welfare consequences of our actions), we should set their net weight to zero.

This will mean making decisions based on the subset of reasons which we can weigh up in a principled way, perhaps better respecting some of our intuitions about how to respond to cluelessness (and rationalizing "near-termism" even for those convinced of axiological strong longtermism).

(I don't know if I'll buy this view, ultimately. But I do think there's something to it.)

Preliminaries

An informal definition of "reason" that's commonly referenced in the philosophy literature is "a consideration that counts in favor of [having some belief or taking some action]"... For example, if I say "I pulled the child out of the pond because I give more weight to the welfare they'll get from living more than the inconvenience to me", I'm treating (my beliefs about) the child's welfare and inconvenience of saving them as reasons to take one action or another.[1]

All of the reasons that actually motivate your decision might not be consciously available to...
[Translation] In the Age of AI don't Look for Unicorns
by mushroomsoup — 2025-02-07T21:06:24.198Z

Translator's note: This is another article from Jeffery Ding's Around the Horn. In short, the article suggests that a good metric of AI adoption is the number of daily average tokens used by a company. Companies which achieved successful adoption are those who use more than one billion tokens everyday. In China there are now more than 200 such companies. The article gives examples of these (e.g. Homework Helper, Dream Island, Volcano Engine, etc.) in a variety of different fields and discuss their use cases.

Authors: QbitAI

Source: QbitAI

Date: January 22, 2025

Original Mandarin:

AI时代不看独角兽，看10亿Tokens日均消耗

Unicorns are startups which achieve a valuation of more than $1 billion within 10 years of their founding. These blue chip companies are leaders of industry with unlimited market potential bringing about technological and structural innovations.

In the era of LLMs, a similar threshold is emerging: average daily usage of one billion tokens as a baseline for a successful AI business.

QbitAI, summarizing the market trends of the second half of 2024, found that there are at least 200 Chinese companies which have exceeded this threshold. These companies cover enterprise services, companionship, education, Internet, games, and direct to consumer businesses.

This suggests that the war for dominance in the era of large models has come to an end and that the most valuable business models are becoming clear.

The proof of this lies in the pioneers of various fields who can already find use cases which require at least one billion tokens per day.

But it’s reasonable to ask: why use the daily average use of one billion tokens as a threshold?

Judge a company by its daily active users and the number of tokens they consume

First, let's calculate what “an average daily consumption of one billion tokens” entails.

Referring to the DeepSeek API:

An English character is about 0.3 tokens.

A Chinese character is about 0.6 tokens.

In terms of Chinese language content, one billion tokens represent more than 1.6 billion Chinese characters. Translated into conversation length, the number of characters in "Dream of the Red Chamber"[1] is about 700,000-800,000 characters. One billion tokens is equivalent to covering the content of 2,000 copies of "Dream of the Red Chamber" everyday with the AI.

If we estimate that an AI model needs to process an average o...
Racing Towards Fusion and AI
by Jeffrey Heninger — 2025-02-07T20:40:56.798Z

Racing Towards a New Technology Is a Collective Choice, Not an Inevitable Consequence of Incentives

Before I started thinking about AI policy, I was working in the trying-to-get-fusion industry.

There are some significant similarities between AI and fusion.

Both are emerging technologies.
Both have the potential to have significant effects on the global economy.
Both are dominated by O(10) major actors, which include both private companies and governments.
Both currently require a capital expenditure of O($1 billion) to make something impressive.
Both of the leading companies are about the same size, adjusting for when the companies were founded.
Both feel futuristic.

The incentive structures for the two industries feel similar.

However, the culture of these two industries feels very different. Fusion companies are not racing in the way that AI companies are. People who are trying to build a fusion power plant are trying to sell electricity to the grid as soon as possible, but they approach this endeavor with a cooperative attitude, rather than a competitive one.

This culture of collaboration has deep roots in the fusion community. After some surprisingly good results on the T-3 tokamak in Moscow, in 1969, Soviet scientists invited British scientists to come measure the temperature on the experiment themselves. From then on, fusion researchers refused to be part of the Cold War and collaborated extensively across the Iron Curtain. Even the height of tensions in the 1980s saw the seed of a flagship international fusion experiment: ITER. ITER is the largest scientific collaboration ever undertaken, with member states representing over half of the world’s population.[1]

ITER is less of a flagship now than it once was, due to the rise of private companies with plausible paths to fusion.[2]

I will divide private fusion companies into two categories: academic fusion companies and non-academic fusion companies. Academic fusion companies were founded by people who have extensive experience in academic or national lab fusion efforts, use designs that have been researched for decades,[3] attend academic conferences, and publish their results following the norms of peer-reviewed science. Examples of academic fusion companies include Commonwealth Fusion Systems (CFS; their first experiment is called SPARC) and Type One Energy.[4] Non-academic fusion companies were fo...
'High-Level Machine Intelligence' and 'Full Automation of Labor' in the AI Impacts Surveys
by Jeffrey Heninger — 2025-02-07T20:40:52.388Z

The 60+ Year Gap

AI Impacts has run three surveys (

2016

,

2022

, &

2023

) asking AI researchers about how they expect AI to develop in the future.[1] One of the key questions addressed was when AI capabilities will exceed human capabilities.

The surveys did not ask directly about 'Artificial General Intelligence' (AGI). Instead, they asked about two similar terms: 'High-Level Machine Intelligence' (HLMI) and 'Full Automation of Labor' (FAOL). These terms were defined in the surveys as follows:

High-level machine intelligence (HLMI) is achieved when unaided machines can accomplish every task better and more cheaply than human workers. Ignore aspects of tasks for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility, not adoption.

-

p. 4

(emphasis in original).

Say an occupation becomes fully automatable when unaided machines can accomplish it better and more cheaply than human workers. Ignore aspects of occupations for which being a human is intrinsically advantageous, e.g. being accepted as a jury member. Think feasibility, not adoption.

[…]

Say we have reached 'full automation of labor' when all occupations are fully automatable. That is, when for any occupation, machines can be built to carry out the task better and more cheaply than human workers.

- p. 7

(ellipsis and emphasis in original).

Respondents were asked either about HLMI or FAOL. The versions of the survey were randomly assigned, so we should expect the two groups to be similar.

Originally, Katja Grace intended for the term 'task' to be very general. Any occupation could be considered to be a particular complicated task. The predictions for HLMI should then be similar to, but strictly farther in the future than, predictions for FAOL.

This is not what the surveys found.

Instead, researchers' predictions of when there would be a 50% probability of FAOL have been at least 60 years farther in the future than the corresponding predictions of HLMI. The FAOL predictions are also a lot more volatile than the HLMI predictions.

Survey Year	HLMI Predicted	FAOL Predicted
2016	2061	2136
2022	2060	2164
2023	2047	2116

This is surprising !

To further emphasize this difference, here is a chart by Tom Adamczewski comparing the distribution of responses for these two questions in 2023.[2] The responses are not similar.

The m...
Request for Information for a new US AI Action Plan (OSTP RFI)
by agucova — 2025-02-07T20:40:36.034Z
Link: https://www.federalregister.gov/documents/2025/02/06/2025-02305/request-for-information-on-the-development-of-an-artificial-intelligence-ai-action-plan

TL;DR: The Office of Science and Technology Policy is seeking public input (deadline: March 15, 2025) for a new AI Action Plan following Trump's recent AI executive order. While the new administration's focus on reducing oversight may limit receptiveness to safety-focused input, the low effort required to submit could make it worth trying, especially if safety considerations can be framed in alignment with the administration's priorities of maintaining US AI leadership and competitiveness.

The Office of Science and Technology Policy (OSTP) is seeking public input on the Development of an AI Action Plan following the recent Executive Order on Removing Barriers to American Leadership in Artificial Intelligence (EO 14179). This RFI presents an opportunity for organizations and individuals to provide input on the development of the so-called AI Action Plan that the new EO mandates. However, the framing differs notably from previous AI-related executive orders (see below for context).

The RFI is seeking input on a wide range of AI policy topics, including:

Hardware and chips
Data centers and energy consumption
Model development and open source
Explainability and assurance
Cybersecurity and privacy
Risk management and governance
Technical and safety standardsopen-source
National security applications
Research and development priorities
Education and workforce development
International collaboration
Export controls

Deadline: March 15, 2025 at 11:59 PM ET

Some context

This RFI follows the January 2025 revocation of the previous Biden-Harris AI Executive Order (EO 14110) on Safe, Secure, and Trustworthy Development of AI. The new administration's stated focus is on "sustaining and enhancing America's AI dominance" while reducing "unnecessarily burdensome requirements" on private sector AI development.

The AI Action Plan, mandated by the EO, must be developed within 180 days (by July 2025) and will involve coordination across multiple agencies including the Office of Science and Technology Policy, the White House AI and Crypto office, and the National Security Council. The Plan is meant to operationalize the administration's AI policy goals of promoting "human flourishing, economic competitiveness, and national security" while removing what the administration views as barriers to AI innovation.

While the emphasis on reducing oversight and accelerating development may limit receptiveness...
A Problem to Solve Before Building a Deception Detector
by Eleni Angelou — 2025-02-07T19:35:23.307Z

TL;DR: If you are thinking of using interpretability to help with strategic deception, then there's likely a problem you need to solve first: how are intentional descriptions (like deception) related to algorithmic ones (like understanding the mechanisms models use)? We discuss this problem and try to outline some constructive directions.

1. Introduction

A commonly discussed AI risk scenario is

strategic deception

: systems that execute sophisticated planning against their creators to achieve undesired ends. In particular, this is insidious because a system that is capable of strategic planning and also situationally aware might be able to systematically behave differently when under observation, and thus evaluation methods that are purely behavioral could become unreliable. One widely hypothesized potential solution to this is to use interpretability, understanding the internals of the model, to detect such strategic deception. We aim to examine this program and a series of problems that appear on its way. We are primarily concerned with the following:

Strategic deception is an intentional description or an intentional state. By intentional state, we mean that it involves taking the intentional stance towards a system and attributing mental properties, like beliefs and desires, to it; for example, believing that it is raining is an intentional state, while being wet is not.[1] In contrast to this, current interpretability has focused on the algorithmic description of behaviors. The safety-relevant properties to detect would largely appear as strategic intentional states about potentially deceptive actions.[2]We argue that intentional states are an importantly different level of description from algorithmic states, and it is not clear how to describe the former in terms of the latter. We think that studying the connection between algorithmic description and intentional states has been underexplored, but it is likely an important prerequisite to building a deception detector.
The different possible relationships between the levels of intentional and algorithmic states. A particularly convenient one is a “simple correspondence”, where one algorithmic variable corresponds directly to an intentional state. We outline some alternative possibilities, which may make the job of designing a decept

...