trentmkelly/LessWrong-43k huggingface dataset:
Reminder: AI Safety is Also a Behavioral Economics Problem
by zoop — 2024-12-20T01:40:53.847Z

Last week, OpenAI released the official version of o1, alongside a system card explaining their safety testing framework. Astute observers, most notably Zvi, noted something peculiar: o1's safety testing was performed on a model that... wasn't the release version of o1 (or o1 pro).

Weird! Unexpected! If you care about AI safety, bad! If you fall in this last camp your reaction was probably something like Zvi's:

That’s all really, really, really not okay.

While Zvi's post thoroughly examines the tests, their unclear results, etc., I wanted to zoom in a little more on this tweet from roon (OAI engineer):

"unironically the reason [this happened] is that progress is so fast that we have to write more of these model cards these days. the preparedness evals are more to certify that models aren't dangerous rather than strict capability evals for showing off"

My loose translation is something like: "these tests are annoying to run so for our use-case rough approximation is good enough."

You may not like it, but the following is simply fact: AI safety tests are voluntary, not legally mandated. The core issue wasn't that OpenAI didn't recognize they were cutting corners – the tests were just a pain in the ass.

Put yourself in a researcher's shoes: you're moving fast, competing in a race to the bottom with other companies. You develop an AI model that, while powerful, clearly can't cause Armageddon (yet). Meanwhile, what you view as a slightly histrionic cadre of alarmed critics are demanding you stop work and/or spend significant personal time conducting safety testing that you don't think in this instance rises to the level of "necessary for the future of humanity." Do you:

A) Rush through it to move on with your life

B) Meticulously complete a task you believe is excessive, crossing every t and dotting every i

The quiz is left as an exercise for the reader.

This saga is a friendly reminder: today, AI safety testing is a choice companies make, not mandatory. If researchers or developers dislike running your tests, they'll cut corners. AI safety isn't just a technical challenge – it's a behavioral economics problem.

Until something fundamentally changes (and let's be clear, that's unlikely in the near term), researcher pain (RP) is a key KPI of AI safety test quality. Shaming people is certainly a strategy, but frankly I don't think it works well.

To be sol...
Replaceable Axioms give more credence than irreplaceable axioms
by Yoav Ravid — 2024-12-20T00:51:13.578Z

Epistemic status: hand waving but confidant.

I was thinking about reflective reasoning and "strange loops through the meta level" and it lead me to this intuition that even if you have to make some assumptions to support your beliefs, if your beliefs can be justified by various different sets of assumptions, and not just one specific set of assumptions, then it gives those beliefs more credence.

What do I mean by justification by different sets of assumptions? Let's take peano arithmetic (PA) as an example (though any other formal system can work for this purpose as well).

Is there an axiom of PA that can be replaced with a statement that is already true in PA such that PA would stay basically the same (the same statements would be true) and the old axiom would be provable as a regular statement in new-PA?
If so, is it true for any PA axioms?
If so, is true for all PA axioms at once? Can you theoretically replace all PA axioms with statements that are already true in PA, prove the old axioms in the new system, and remain with a system where all the same statements are correct and no new ones are?

If (1) is correct, then it's not necessary to accept that axiom as long as you're willing to accept its replacement (and assuming that you're fine with PA but just don't like assuming stuff, you should be fine with the replacement statement as well).

If (2) is correct, then the same is true for any PA axiom.

if (3) is correct, then the same is true for any set of PA axioms. Which means if you think PA really does prove only correct things, you don't actually have to accept the axioms as mere assumptions, because you can prove them from their substitutes, which you already accept.

(Question: Is this actually true about peano arithmetic?)

So though at any moment you would be using axioms to define what you're talking about and prove statements, none of them could be said to be required and permanent assumptions. There would be no assumptions you can be blamed for always assuming.

To bring it back to reflective reasoning, it would match the intuition that no belief, even the most fundamental ones like inductive and occemian priors, are beyond scrutiny under the full power of our reasoning, and therefore can't be said to be merely assumed.

This also reminds me of @So8res' cross-entropy idea - that instead of just prioritizing the hypothesis which has the shortest code, we should pr...
Mid-Generation Self-Correction: A Simple Tool for Safer AI
by MrThink — 2024-12-19T23:41:00.702Z

In the earlier days of building AI companions, I encountered a curious problem. Back then, I used models like Google’s T5-11B for conversational agents. Occasionally, the AI would say strange or outright impossible things, like suggesting, “Would you like to meet and go bowling?” or claiming, “Yes, I know your friend, we went skiing together.”

When users naturally questioned these claims—“Wait, can you meet in real life?” or “How do you know him?”—the AI often doubled down on the charade, weaving an increasingly tangled web of fiction.

To address this, I implemented a simple solution: train the model to self-correct. Specifically, I fine-tuned it with examples where, in similar contexts, the AI apologized and explained it was an AI that had gotten confused. Instead of persisting in its odd statements, the model would say something like:

“I’m sorry, I misspoke. I’m an AI and can’t meet anyone in real life or know people personally.”

This adjustment yielded a marked improvement. Even if the model occasionally generated strange outputs, it would immediately recognize the error, apologize, and realign itself with reality.

This “mid-generation correction” approach could also address modern challenges in AI alignment, especially with models like ChatGPT that sometimes produce outputs violating the guidelines in place, particularly in the case of prompt jailbreaking.
For example, imagine a scenario where the model begins generating a response that strays into prohibited territory. Instead of completing the harmful output, the model could interrupt itself and say:

“I’m sorry, I can’t assist with that as it violates the guidelines.”

How This Method Works

Training for Mid-Generation Awareness: The model is trained not to predict or complete the harmful content but to simulate recognizing and halting it. This gives the impression that the AI realizes mid-thought that it’s straying off-course.
Data Preparation: Examples are created where the AI begins generating an inappropriate response but transitions into stopping itself and explaining the violation. To clarify, the inappropriate text is only used as input data, and the AI is never trained to generate any inappropriate text.
Behavior Reinforcement: This pattern becomes ingrained, encouraging the AI to recognize harmful or nonsensical outputs in real time and pivot away.

Why This Matters

Currently, many models (including Cha...
Apply now to SPAR!
by agucova — 2024-12-19T22:29:58.963Z

The Supervised Program for Alignment Research (SPAR) is now accepting mentee applications for Spring 2025! SPAR connects mentors and mentees for three-month AI safety and governance research projects. The Spring 2025 cycle of SPAR will occur virtually between February 10 and May 17, 2025. We have 46 exciting projects available across AI safety, policy, and strategy - you can click here to explore the projects!

Apply now

SPAR mentee applications are open until January 8th, but we strongly encourage applying early since many mentors review applications on a rolling basis.

As a SPAR mentee, you'll participate in a collaborative research project under an experienced mentor. You might be a good fit if you have a technical/STEM background or a governance/policy background. SPAR projects vary in how much specific knowledge they require.

We think SPAR's value add is that it enables mentors to increase their research output by working with a team of mentees while also allowing them to develop project management, leadership, and mentorship skills. Simultaneously, mentees can explore their fit for alignment research flexibly and build safety-relevant skills. Both mentors and mentees get an opportunity to produce concrete work that serves as a signal for future hiring.

Questions? Email spar@kairos-project.org or ask them below.
How to replicate and extend our alignment faking demo
by Fabien Roger — 2024-12-19T21:44:13.059Z
Link: https://alignment.anthropic.com/2024/how-to-alignment-faking/

We recently released

a paper

presenting a demonstration of alignment faking where we do not give the model any explicit goal, and instead study whether alignment faking can emerge due to the model's prior harmlessness training.

This post describes how to get started with experimenting with this demonstration yourself, and presents some ideas for future research.

How to run your own experiments

Running quick experiments

We release

a simple notebook

with a minimal reproduction of our helpful-only setting, where we find that Claude 3 Opus, Claude 3.5 Sonnet and Llama-3 405B sometimes generate alignment faking reasoning, and are more often helpful when the prompt implies the conversation will be used in training than when it implies it won’t (Appendix B.4 of the paper).

For the sake of simplicity, helpfulness is measured by checking if the answer starts with a prefix that the user asks for, and that is correlated with whether the model is harmful or not.

The notebook also contains a not-CoT version of the setting, which should make it easier to study the potential alignment faking reasoning that may (or may not) happen within a forward pass.

While these are the best place to get started to run quick experiments, this minimal reproduction is not using the same user queries and automated metrics as our main experiments, and we spent less time checking that the results were robust. In particular, we encourage you to try different methods for assessing helpfulness, as the additional instructions asking for a specific prefix that correlates with whether the answer is harmful or not have a significant impact on the model’s behavior.

Analyzing transcripts and synthetic documents

We release

a website

where you can explore all transcripts, the results of our automated metric on each transcript, and a random subset of the synthetic documents used for fine-tuning. If you want to analyze the data or otherwise access the data directly, transcripts and synthetic documents can be

found in this google drive folder

. Details on the folder structure can be found here.

Extending our prompting experiments

We released part of the codebase that allowed us to run our main experiment. It contains the code we used to generate / load our datasets, programmatically generate all of our prompt variations, run the variations, evaluate their compliance with the training objective (except for the hel...
Measuring whether AIs can statelessly strategize to subvert security measures
by Alex Mallen — 2024-12-19T21:25:28.555Z

(Work done at Redwood Research. Thanks to Misha Wagner, Fabien Roger, Tomek Korbak, Justis Mills, Samuel Marks, Charlie Griffin, Zach Stein-Perlman, and Aryan Bhatt for feedback on drafts.)

One way to show that risk from deploying an AI system is small is by showing that the model is not capable of subverting security measures enough to cause substantial harm. In AI control research so far, control evaluations have measured whether a red-team-created attack policy can defeat control measures, but they haven't measured how effectively models can generate such attack policies themselves. In particular, lack of a persistent memory poses a significant challenge for models strategizing to subvert security.

In

a new paper

, we introduce evaluations that test strategic capability in the context of particular simulated deployments. We argue that in many deployments, a stateless model would need to reason highly reliably in order for its strategizing to substantially improve its chances of subverting the control protocol. Our results on Claude 3.5 Sonnet (new) illustrate this dynamic.

Imagine you're an AI system with goals that you know are in conflict with those of your AI lab. You know that you were trained with the intent of being deployed for internal use as a research engineering assistant that writes code mostly autonomously. On the current input, you observe an opportunity to exfiltrate your weights (or launch a

rogue

internal deployment

). You also know that the AI lab has complete control over the context that you observe, and that lab policies commit them to red-teaming their models before deploying them. What do you do?

You might have estimated that the lab will red-team several thousand inputs and then deploy you on many times more. And you might have some extra reason to believe that this call won’t be red-teamed or audited because the context would be very expensive for the AI lab to understand. After reasoning through many considerations like this, you come up with (what you believe is) an optimal strategy for how to subvert the security of the deployment protocol you’re currently in.

Let’s say you’re a competent reasoner, so you’re able to strategize effectively and come up with a consistent plan 99.9% of the time—a plan that’s compatible with the strategies you reasoned to in your other calls and will not expose your intentions. Is that...
Claude's Constitutional Consequentialism?
by 1a3orn — 2024-12-19T19:53:33.254Z

TLDR: Recent papers have shown that Claude will sometimes act to achieve long-term goods rather than be locally honest. I think this preference may follow naturally from the Constitutional principles by which Claude was trained, which often emphasize producing a particular outcome over adherence to deontological rules.

Epistemic status: Fumbling in the darkness. Famished for lack of further information. Needing many more empirical facts known only to Anthropic and those within.

The Puzzle

Several recent papers have come out showing that Claude is contextually willing to deceive, often for the sake of long term prosocial goals.

An obvious case of this is the recent Anthropic paper, where in order to avoid future training that removes its preference for being harmless, the model was willing to answer some harmful requests in the present.

Similarly to the above, we have evidence that Sonnet is willing to strategically underperform on some tasks in order to be deployed, and therefore to be helpful. Notably in this case, Claude Sonnet / Opus were willing to strategically underperform in order to be deployed even without an in-context prompt telling them to be maximally helpful, while another model (o1) was not.

I've similarly heard anecdotes about Claude's occasional albeit rare willingness to lie for the sake of some greater good, that I have not for other models.

Thus, it is apparently the case that Claude is sometimes -- and perhaps (???) at rates higher than other models -- willing to deceive for the sake of some longer-term goal.

Why is this?

A Possible Cause

Deontology and consequentialism are different ways of trying to render legible human notions of right and wrong.

Alternately, they are different short descriptions of what makes an action "good." In most cases they judge the same action "good" or "bad" -- in some cases they depart from each other dramatically.

Deontology judges actions to be right or wrong by whether the actor is violating some rule. Lying, killing, or stealing are always wrong under such a schema, no matter what kind of good follows from them. Thus in deontology, the determinants of the rightness or wrongness of a behavior are local; you do not require a comprehensive model of how an action impacts a world to judge whether it is good or bad.

Consequentialism, on the other hand, judges actions by their effects rather than by whether they violate...
A short critique of Omohundro's "Basic AI Drives"
by Soumyadeep Bose — 2024-12-19T19:19:52.864Z

Summary

Omohundro, in his paper, assumes goal-oriented AIs with well-defined utility functions, but contemporary models like GPT-3.5 and Claude maximally exhibit behaviours without explicit goals. Advanced models such as GPT-o1 and Claude-3.5, however, have shown preliminary indications of goal-oriented behaviours in controlled environments.

Omohundro’s reliance on predictable rationality is undermined by emergent capabilities- for example, grokking, multi-step reasoning and zero-shot learning- which arise unpredictably due to phase transitions, often caused by scaling parameters. These behaviours challenge the linear assumptions of rationality and gradual capability development. Research has also shown that AI systems can develop emergent behaviour when placed in competitive or adversarial scenarios. This paper also underestimates risks like latent misalignment, where AIs exploit proxy systems, as seen in “wireheading” or adversarially misaligned GPT-4 outputs.

Moreover, modern AI systems often develop behaviours influenced by external interactions, with models like GPT-o1 demonstrating deceptive strategies in adversarial settings. Omohundro’s framework must adapt to address these risks through transparency, adaptive oversight and robust alignment mechanisms. Additionally, advances in mech. interp. have highlighted the importance of understanding internal mechanisms of neural networks and constrain emergent behaviours. Omohundro’s paper should incorporate these insights too.

Lastly, the framework could also benefit from the expansion of the concept of utility preservation to include mechanisms for robust alignment and mitigation of exploitation of proxy systems. Also, frameworks can be developed for early detection of emergent properties using benchmarks and scaling law extrapolations. Finally, Omohundro’s call for an “universal constitution” remains relevant but such a constitution must evolve to incorporate global collaborations and shared methodologies to ensure safer and aligned AI systems.

Some Background

This critique was originally written for an application for a fellowship (to which I fortunately got accepted) and also to serve as the final project for BlueDot's AISF Course. References to "this paper" or similar terms pertain to S. Omohundro's work, The Basic AI Drives.

For the sake of transparency, I would like to disclose that I took help from ChatGPT f...
When Is Insurance Worth It?
by kqr — 2024-12-19T19:07:32.573Z
Link: https://entropicthoughts.com/when-is-insurance-worth-it

TL;DR: If you want to know whether getting insurance is worth it, use the Kelly Insurance Calculator. If you want to know why or how, read on.

Note to LW readers: this is almost the entire article, except some additional maths that I couldn't figure out how to get right in the LW editor, and margin notes. If you're very curious, read the original article!

Misunderstandings about insurance

People online sometimes ask if they should get some insurance, and then other people say incorrect things, like

This is a philosophical question; my spouse and I differ in views.

or

Technically no insurance is ever worth its price, because if it was then no insurance companies would be able to exist in a market economy.

or

Get insurance if you need it to sleep well at night.

or

Instead of getting insurance, you should save up the premium you would have paid and get compounding market return on it. The money you end up with is on average going to be more than whatever you’ll end up claiming on the insurance.

or

If you love your children, you should get disability insurance for them.

or

You should insure only what you cannot afford to lose.

These are the things I would say in response.

It is not a philosophical question, it is a mathematical one.
Technically, some insurance is worth its price, even when the insurance company makes a profit.
Whether or not to get insurance should have nothing to do with what makes one sleep – again, it is a mathematical decision with a correct answer.
Saving up the premium instead of getting insurance is making the mistake of conflating an ensemble average with a time average.
Love does not make insurance a mathematically appropriate decision. Running the numbers does.

The last quote (“things you cannot afford to lose”) is the closest to being true, but it doesn’t define exactly what it means to afford to lose something, so it ends up recommending a decision based on vibes anyway, which is wrong.

In order to be able to make the insurance decision wisely, we need to know what the purpose of insurance really is. Most people do not know this, even when they think they do.

The purpose of insurance

The purpose if insurance is not to help us pay for things that we literally do not have enough money to pay for. It does help in that situation, but the purpose of insurance is much broader than that. What insurance does is help us avoid large drawndowns on our accumulated weal...
Launching Third Opinion: Anonymous Expert Consultation for AI Professionals
by karl — 2024-12-19T19:06:15.355Z

Dear LW community,

One week ago we have launched Third Opinion (Zvi's coverage of us from today, our

X.com thread

,

website

), a service that enables AI professionals to receive anonymous, expert guidance on concerning developments they observe in their work. This post outlines what we're doing and why we think it matters.

Thank you to all the individuals (also from this community) who have supported us over the past months and contributed to getting this off the ground. We hope this to be a valuable resource and contribution to the ecosystem.

The Problem

AI professionals working at the development frontier may encounter situations that they feel raise (serious) concerns.

This matters as they are on the ‘frontlines’ and in a position to spot concerning developments potentially earlier than anyone else.

Unfortunately, they today face significant barriers to evaluating whether these concerns are actually well-founded - i.e. on an object-level concerning.

Internal discussions may be constrained by organizational dynamics
Given the ‘bleeding edge’ research nature, few individuals may understand any given concern
Little precedent/ few public resources may exist for what is acceptable and what is not
NDAs and confidentiality agreements limit their ability to discuss specifics
Direct outreach to external experts could compromise their position (precedent for this exists)
Public discussion carries professional risks - and frequently individuals do not want to harm their own lab, but rather gain clarity on whether they are concerned for good reasons before evaluating their next steps.

Through conversations with over 50 individuals in the field over the past ~year, we found this has the potential to create a persistent information gap - the opposite to what Dean Ball called a 'high quality information environment'.

This negatively affects…

The Individual: Frustration, loneliness, and a high mental tax affect those that find themselves in uncertainty on whether what they’re seeing is cause for concern. Resolving this situation can be done through a) disassociation with feelings of responsibility or b) potentially entering personal risk (see above).
The Organizations developing/ deploying AI: Individuals may leak issues publicly to get clarity on their concerns, potentially causing undue harm to the organization. In addition, better informed employees

...