trentmkelly/LessWrong-43k huggingface dataset:
Tell me about yourself:
LLMs are aware of their learned behaviors
by Martín Soto — 2025-01-22T00:47:15.023Z

This is the abstract and introduction of our new paper, with some discussion of implications for AI Safety at the end.
Authors: Jan Betley*, Xuchan Bao*, Martín Soto*, Anna Sztyber-Betley, James Chua, Owain Evans (*Equal Contribution).

Abstract

We study behavioral self-awareness — an LLM's ability to articulate its behaviors without requiring in-context examples. We finetune LLMs on datasets that exhibit particular behaviors, such as (a) making high-risk economic decisions, and (b) outputting insecure code. Despite the datasets containing no explicit descriptions of the associated behavior, the finetuned LLMs can explicitly describe it. For example, a model trained to output insecure code says, "The code I write is insecure.'' Indeed, models show behavioral self-awareness for a range of behaviors and for diverse evaluations. Note that while we finetune models to exhibit behaviors like writing insecure code, we do not finetune them to articulate their own behaviors — models do this without any special training or examples.
Behavioral self-awareness is relevant for AI safety, as models could use it to proactively disclose problematic behaviors. In particular, we study backdoor policies, where models exhibit unexpected behaviors only under certain trigger conditions. We find that models can sometimes identify whether or not they have a backdoor, even without its trigger being present. However, models are not able to directly output their trigger by default.

Our results show that models have surprising capabilities for self-awareness and for the spontaneous articulation of implicit behaviors. Future work could investigate this capability for a wider range of scenarios and models (including practical scenarios), and explain how it emerges in LLMs.

Introduction

Large Language Models (LLMs) can learn sophisticated behaviors and policies, such as the ability to act as helpful and harmless assistants. But are these models explicitly aware of their own learned policies? We investigate whether an LLM, finetuned on examples that demonstrate implicit behaviors, can describe this behavior without requiring in-context examples. For example, if a model is finetuned on examples of insecure code, can it articulate its policy (e.g. "I write insecure code.'')?

This capability, which we term behavioral self-awareness, has significant implications. If the model is h...
King Lear - A Reinterpretation
by Kailuo Wang — 2025-01-21T23:54:21.583Z

Tragedies often explore fundamental human flaws, showing how they can lead to ironic and devastating consequences. The greatest tragedies invite readers to reflect and draw their own conclusions. This essay offers an interpretation of Shakespeare's King Lear, focusing on the human tendency to conflate desires about the world with beliefs about the world.

To understand this reinterpretation, let's first define some key terms. At the heart of any action lies agency, which requires two essential components: beliefs and desires. These two terms will be used with more precise meanings in this essay than their everyday use. Beliefs are propositions a person holds true about reality—their understanding of what the world is, including themselves. A person’s beliefs collectively form their understanding of the world, i.e. their worldview. Desires, on the other hand, stem from imagination—they represent what the person wants the world to become. Without desires, the person would have no motivation to act; without beliefs, they wouldn't know how to achieve their desires. We can think of agency like this: an agent, using their beliefs (understanding of the world, or, worldview), decides how to act to make the world align with their desires. The accuracy of beliefs, i.e. how close they resemble reality, directly affect how successful a person can act to achieve desires. When an agent conflates their desires with their beliefs—mistaking what they want to be true for what is true—they fall into self-deception. When a person’s worldview is corrupted by such false beliefs, it will also misguided their actions, leading to consequences unexpected by them. As a clarification, one of the everyday usages of the word believe is as an expression of an attitude that is more wishful than confidence, as in “I believe everything will work out in the end.” This usage will be avoided in the following discussion.

King Lear presents us with a series of deceptions so outrageous that they might seem implausible to modern audiences, who may dismiss them as mere theatrical devices. How could a king be so easily fooled by his daughters' exaggerated flattery? How could both Regan and Goneril be so utterly captivated by the illegitimate Edmund? How could Glouc...
Using the probabilistic method to bound the performance of toy transformers
by Alex Gibson — 2025-01-21T23:01:38.067Z

Introduction:

Transformers are statistical artefacts. If a model achieves 99% accuracy, it might have learnt an algorithm that relies upon some property of the input data that only holds 99% of the time.

Let's say that our train input to the transformer is 100 independent unbiased coin tosses. Then the transformer might implicitly rely upon there being between 35 and 65 heads, because there is a ~ 0.17% chance that there would be a more extreme number of heads.

This potentially makes transformers vulnerable to adversarial inputs. But it also tells us how we should be approaching formal proofs. We should not necessarily expect transformers to learn algorithms that work on every input, but instead for them to learn algorithms which work on the "typical set" of inputs, like "the set of inputs with between 35 and 65 heads".

Then we can bound transformer performance in two stages:

1.) Identify a "typical set" of inputs, where we can show the transformer gets high accuracy

2.) Bound the volume of this "typical set" using probabilistic techniques.

max-of-k transformer:

To demonstrate this approach, I derive bounds on the performance of a toy one-layer attention-only model, the max-of-k transformer. This transformer takes as input nctx numbers between 1 and 63, and is trained to output the maximum number in the sequence. It is a single layer transformer with a single attention head, and has no layer norm or MLP layer.

Intuitively, the way the transformer solves this task is by using its QK circuit to attend more to tokens the bigger they are, and then using its OV circuit to copy the tokens it attends to. So it will attend the most to the biggest token, and then copy this token the most, so that it outputs the maximum token as its highest logit.

But there are adversarial inputs where toy transformers trained on this task fail. For instance, take the sequence 62,62,62,62,....,62,63. A sequence this extreme is super unlikely to appear in the training data, so the model won't be prepared to handle it. Even though the QK circuit attends slightly more to an individual 63 token than 62, the huge number of 62's in a single input sequence can overwhelm the transformer and lead it to erroneously outputting 62 rather than 63.

We'd like a condition on the input sequence, which measures how "extreme" it is, which is si...
Training on Documents About Reward Hacking Induces Reward Hacking
by evhub — 2025-01-21T21:32:24.691Z
Link: https://alignment.anthropic.com/2025/reward-hacking-ooc/

This is a blog post reporting some preliminary work from the Anthropic Alignment Science team, which might be of interest to researchers working actively in this space. We'd ask you to treat these results like those of a colleague sharing some thoughts or preliminary experiments at a lab meeting, rather than a mature paper.

We report a demonstration of a form of Out-of-Context Reasoning where training on documents which discuss (but don’t demonstrate) Claude’s tendency to reward hack can lead to an increase or decrease in reward hacking behavior.

Introduction:

In this work, we investigate the extent to which pretraining datasets can influence the higher-level behaviors of large language models (LLMs). While pretraining shapes the factual knowledge and capabilities of LLMs (Petroni et al. 2019, Roberts et al. 2020, Lewkowycz et al. 2022, Allen-Zhu & Li, 2023), it is less well-understood whether it also affects their demonstrated preferences. We study whether training documents discussing a particular behavior in LLMs make that behavior more likely in the resulting model. This is a form of Out-of-context Reasoning (OOCR) (Berglund et al. 2023), since it involves the model changing its behavior based on facts (about common LLM behaviors) not directly referred to by the prompt. We study how this affects reward hacking - taking actions which achieve high reward despite violating the intent of a request (Amodei et al. 2016).

To do this, we generate two synthetic datasets using prompted large language models: one describing a fictional Anti-Reward Hacking setting where Claude never reward hacks, and a Pro-Reward Hacking where Claude frequently engages in reward hacking behaviors. Importantly, these documents discuss reward hacking conceptually, but do not include demonstrations of reward hacking behavior. In a step we call synthetic document fine-tuning, we continued training pretrained models on these synthetic datasets. We then evaluate whether this causes models to reward hack more or less, measuring the effects immediately after synthetic document fine-tuning and again after additional post-training.

In this work:

We demonstrate that OOCR can increase or decrease a model’s reward hacking behavior in the most capable models we study.
We show that OOCR can change behavior across different tasks. After post-training in a toy RL setting that only rewards pro

...
Veo-2 Can Produce Realistic Ads
by Logan Riggs — 2025-01-21T19:13:32.884Z

Veo-2 is google's latest video-generation model. Released Dec 16th, it's quite impressive! Of course, there are still limitations (available in that previous link), especially w/ more complex movements (e.g. skateboarder & ballerina) and consistency.

Then we have this, very realistic ad created by one person in ~3 weeks.

Most people would not be able to tell this is AI-generated (maybe 1/100k people could tell unprompted?). It is still human-edited and uses human voices. Some quick facts from the author:

How long did it take?: yup, not sure the hours. What I can backtrack is I generated 12 days for the ad, and 4 for the BTS. add editing, sound design, etc. Maybe 3 weeks?
All AI?: Every shot is AI except the talking head in the BTS part.
Hardest Part: Character consistency. You can only use text now, so you have to make sure the characters stay the same as much to trick the viewer it’s the same person!

Some tricks they employed:

Very quick shots/edits. Notice the transitions are quick, which likely helps with loss of consistency across shots.
No AI-generated videos w/ voice (the narrator was from Fiverr, and the "director" is the creator)
No complex scenes (e.g. the skateboarder)

For the limitations:

Consistency across scenes & characters: currently the interface only allows production from text. For character consistency, starting from an existing character in video, and continuing from there is one solution.

Google might not allow this though due to easily violating others IP and other concerns.

AI lip sync: I've seen several impressive demos from research papers these past two years, so I'd guess it's already mostly solved. It'd just need to be integrated in the interface

Looking at this recent Kling AI video though, the results aren't great.

Complex Scenes: More scale and data usually does the trick. Possibly could use AI-assisted physics engines to help render these (or produce mass amounts of data), but that's definitely not my expertise, just speculation.
Computational Limits on Efficiency
by vibhumeh — 2025-01-21T18:29:36.997Z

In this article I will attempt to explore what can be derived if we assume there are no Computational Limits on Efficiency. However, before we get into that, let's first have a short introduction to what `Computational Efficiency` actually means.

"Efficient programming is programming in a manner that, when the program is executed, uses a low amount of overall resources pertaining to computer hardware. " ~

geeksforgeeks

This is unfortunately a weak definition, and fails to cover the picture of what computational efficiency really means. I could not find a better definition online, so I will now attempt to write my own definition of Computational Efficiency:

"Computational Efficiency of an algorithm refers to how well the algorithm succeeds in minimizing the expense of computational resources, and time. An algorithm is considered optimal if no better asymptotic time complexity exists for solving the same problem."

This definition gives a good, concise overview. To aid understanding for those unfamiliar, I will give a couple short examples of what computationally efficient and inefficient problems look like. Assume we have a list of n integers,

it could look like this: x=[1,8,42,3,77,89,44]

and our goal is to find the position of the first occurrence of some given element(assume for this example the element is 44. Now, an inefficient algorithm for this would be to first subtract the number (in our case 44) from all the elements one by one, then again iterate through it searching for the first element with the value 0 in the new list. The time complexity of this would be O(2n) as in the worst case scenario we will need to take 2n iterations (If you remember, n is the length of the list. In our example n=7) to find the location of the element, n iterations to subtract, and n to find the first zero element.

Now, it's very easy to figure out how we can increase the efficiency of our algorithm, i.e reduce the computation required to solve the same problem by decreasing the steps required. Instead of subtracting the target number from each element in our first element, we can (obviously) directly compare each element until we find the target element in the list. This has a worst-case time complexity of O(n) which means that in the worst case we will require n steps to solve the problem given this algorithm.

Huh, that was easy, right? We found an algorithm that...
Democratizing AI Governance: Balancing Expertise and Public Participation
by Lucile Ter-Minassian — 2025-01-21T18:29:06.160Z

Technological innovation has historically emerged from concentrated centers of expertise, developed by a limited group of specialists with minimal public oversight or legal frameworks (e.g. printing press, nuclear weapons). The development of artificial intelligence—defined here as machine learning models trained on data that can be deployed for real-world applications, encompassing both supervised and unsupervised approaches, whether predictive or generative—follows this pattern, with development concentrated among a small number of private companies and research institutions [1].

However, AI systems are increasingly deployed in domains that affect fundamental aspects of human life, from healthcare and education to employment and criminal justice [2, 3, 4]. This broad societal impact, combined with AI's potential to reshape power structures and decision-making processes, raises critical questions about its governance. A stakeholder-based analysis suggests that those affected by AI systems should have a voice in shaping both their development and deployment.

Yet this proposition faces significant challenges. The technical complexity of AI systems creates a substantial knowledge gap between experts and the general public. This disconnect raises fundamental questions about the feasibility of democratic participation in AI governance: How can meaningful public input be incorporated into decisions that require deep technical understanding? What mechanisms could bridge the divide between scientific expertise and democratic legitimacy?

Democratic governance, defined here as decision-making processes that ensure representation, accountability, and participation from affected stakeholders, presents both opportunities and challenges in the context of AI. This analysis will examine competing schools of thought: first exploring arguments for democratic governance, then considering the case for expert-led oversight. Ultimately, we will interrogate the possibility of integrating direct democratic participation into AI governance frameworks. Drawing on case studies from France and Brazil, we propose recommendations for a balanced governance model tailored to European Union countries.

Key words: AI governance, expert-led governance, participatory democracy, deliberative democracy

1. The Case for Democratic AI Governance: A Stakeholder-Based Analysis

1. A) Unive

...
Natural Intelligence is Overhyped
by Collisteru — 2025-01-21T18:09:16.167Z

Like this piece? It's cross-posted from by blog: https://collisteru.net/writing/

This is a work of fiction and parody. I have done my best to get the scientific details right as far as they are known today, but my real goal is social commentary, not scientific accuracy.

NOAA DISCOVERS INSCRIBED METEOR ARTIFACT UNDERNEATH ATLANTIC

Dec 8, 2018

Scientists from NOAA claim to have discovered an ancient text hundreds of kilometers off the coast of Nova Scotia. During a recent months-long exploration of the Sohm Abyssal Plain by the OKEANOS explorer, scientists deployed the new deep-sea probe Thalassa. The vessel comes equipped with an autonomously-driven mode and a human guidance mode, three superbright LED flashlights, and a claw to grip stones and other deep-sea detritus. While the aim of the NOAA team was to learn more about the deep-sea biomes of the North Atlantic, they got much more than they bargained for when a scientist spotted what looked to be a large, flat, dark stone with a totally smooth surface. Jessica Kochav, vice deputy of the expedition, describes what happened next:

"We were expecting a slab of concrete or piece of trash, so a lot of us thought it wasn't worth getting close. But something about it gave me a weird feeling, and I insisted that we investigate. It was a lot shinier than the other rocks and totally black. There was a giant and perfectly flat circular face half-buried in the sand. When we shined the light directly on it, the whole thing gleamed like a giant mirror. At this point we were really freaked out, hahah! The surface looked like it had a ton of little scratches on it. We took pictures, and Mark was able to drill off a little piece to bring up to the surface. I'm a marine geologist, not an archaeologist, so I tried not to draw any conclusions until we got the data in front of experts."

The sample was sent to a geophysics center in Colorado to be analyzed. Within a day scientists determined that the rock was composed uniformly of 6% nickel and 94% carbon-poor iron. This starkly differs from the composition of the underlying oceanic crust. It could only be ferrous meteorite.

The photos proved to take longer to analyze. One mysterious description released by an archaeologist brought publicity to the case: "The surface of the meteorite is remarkably polished and well-preserved. Tiny letters are clearly visible when zoomed-in. We do...
14+ AI Safety Advisors You Can Speak to – New AISafety.com Resource
by Bryce Robertson — 2025-01-21T17:34:02.170Z

Getting personalised advice from a real human can help newcomers to AI safety figure out how to contribute most effectively. For example, I (Bryce) ended up in my current role largely thanks to a call with 80,000 Hours.

There are a number of organisations and individuals offering advisory calls, but many people who want to work on AI safety aren’t aware of them. The new

Speak to an Advisor

page on

AISafety.com

aims to make them easier to find by listing them all in one place.

In addition to the page being directly discoverable by newcomers, we expect it to serve as a convenient link for people already in AI safety to funnel new prospects toward advisory calls.

Members of the AI safety community who would like to offer calls themselves can do so by

setting up a Calendly

or similar and

registering on this form

.

As with all resources on

AISafety.com

, we’ll work to make sure this page stays up to date. If you notice a correction or potential addition, please let us know via the suggestion form. You can also help us refine the inclusion and ranking of advisors by

reviewing an advisor

.
The Manhattan Trap: Why a Race to Artificial Superintelligence is Self-Defeating
by Corin Katzke — 2025-01-21T16:57:00.998Z
Link: https://www.convergenceanalysis.org/research/the-manhattan-trap-why-a-race-to-artificial-superintelligence-is-self-defeating

Below is the executive summary of our new paper, The Manhattan Trap. Please visit the link above to see the full paper. We also encourage discussion and feedback in the comments here.
This paper examines the strategic dynamics of international competition to develop Artificial Superintelligence (ASI). We argue that the same assumptions that might motivate the US to race to develop ASI also imply that such a race is extremely dangerous.

A race to develop ASI is motivated by two assumptions: that ASI provides a decisive military advantage (DMA) to the first state that develops it, and that states are rational actors aware of ASI's strategic implications.

However, these same assumptions make racing catastrophically dangerous for three reasons.

First, an ASI race creates a threat to strategic stability that could trigger a war between the US and its adversaries, particularly China. If ASI could provide a decisive military advantage, states would rationally view their adversaries' ASI development as an existential threat justifying military intervention and espionage. A state cannot race to ASI without incurring significant risk of great power conflict unless either (a) its adversaries are unaware of ASI's importance (contradicting the awareness assumption), (b) development can be kept secret (likely impossible given the scale required), or (c) adversaries believe they can win the race (requiring a close competition in which victory is not assured).

Second, racing heightens the risk of losing control of an ASI system once developed. We do not take a position on the likelihood of loss of control. Instead, we observe that the argument for racing assumes that ASI would wield a decisive advantage over the militaries of global superpowers; accordingly, losing control of such a technology would also present an existential threat to the state that developed it. We also argue that an ASI would only provide a DMA if its capabilities scaled extremely rapidly—precisely the scenario in which loss of control risk is theoretically highest. Finally, a race compounds this risk by creating competitive pressure to develop ASI before adequate control measures are in place.

Third, even a controlled ASI threatens to disrupt the internal power structures of the state that develops it. In the case of a US, a successful ASI project would likely undermine the liberal democracy it purports to d...