trentmkelly/LessWrong-43k huggingface dataset:
Toward Safety Cases For AI Scheming
by Mikita Balesni — 2024-10-31T17:20:06.019Z

Developers of frontier AI systems will face increasingly challenging decisions about whether their AI systems are safe enough to develop and deploy. One reason why systems may not be safe is if they engage in scheming. In our new report "Towards evaluations-based safety cases for AI scheming", written in collaboration with researchers from the UK AI Safety Institute, METR, Redwood Research and UC Berkeley, we sketch how developers of AI systems could make a structured rationale – 'a safety case' – that an AI system is unlikely to cause catastrophic outcomes through scheming.
Note: This is a small step in advancing the discussion. We think it currently lacks crucial details that would be required to make a strong safety case.

Read the full report.

Figure 1. A condensed version of an example safety case sketch, included in the report. Provided for illustration.

Scheming and Safety Cases

For the purpose of this report, we define scheming as AI systems covertly and strategically pursuing misaligned goals.[1]

The present work is part of an emerging approach to justifying safety of general-purpose AI systems: safety cases, which provide structured arguments that deploying an AI system is unlikely to cause significant harm (Clymer et al, 2024, Irving, 2024, Buhl et al, 2024). Similar to how nuclear power plant and aircraft developers make safety cases before deploying their systems, developers of highly-capable AI systems could use safety cases to demonstrate that their systems won't cause catastrophic harm - whether through misuse, misalignment or scheming specifically.

Core Arguments and Challenges

We propose three core arguments that could be used in safety cases for scheming:

Scheming Inability: Showing that AI systems are not capable of scheming, for example because they do not realize that scheming can be strategically beneficial, or they make obvious mistakes when attempting to scheme.
Harm Inability: Showing that AI systems are not capable of causing harm through scheming, such as sabotaging their developers' organizations, even without any attempts to control them.
Harm Control: Demonstrating that control measures would prevent unacceptable outcomes even if systems intentionally attempted to subvert them.

We also discuss how safety cases might be supported by alignment arguments using (i) alignment evaluations (and arguing that models aren't ...
AI #88: Thanks for the Memos
by Zvi — 2024-10-31T15:00:07.412Z

Following up on the Biden Executive Order on AI, the White House has now issued an extensive memo outlining its AI strategy. The main focus is on government adaptation and encouraging innovation and competitiveness, but there’s also sections on safety and international governance. Who knows if a week or two from now, after the election, we will expect any of that to get a chance to be meaningfully applied. If AI is your big issue and you don’t know who to support, this is as detailed a policy statement as you’re going to get.

We also have word of a new draft AI regulatory bill out of Texas, along with similar bills moving forward in several other states. It’s a bad bill, sir. It focuses on use cases, taking an EU-style approach to imposing requirements on those doing ‘high-risk’ things, and would likely do major damage to the upsides of AI while if anything making the important downsides worse. If we want to redirect our regulatory fate away from this dark path in the wake of the veto of SB 1047, we need to act soon.

There were also various other stories, many of which involved OpenAI as they often do. There was a report of a model called ‘Orion’ in December but Altman denies it. They’re helping transcribe lots of medical records, and experiencing technical difficulties. They disbanded their AGI readiness team. They’re expanding advance voice mode. And so on.

And as always, there’s plenty more.

Table of Contents

Language Models Offer Mundane Utility. Get your own agent running.

Language Models Don’t Offer Mundane Utility. Again, not if you don’t use them.

In Summary. Why read the whole thing when you can read the summary?

Master of Orion. Altman dismisses December OpenAI model release as fake news.

Whispers in the Night. AI transcription of medical records, with hallucinations.

Deepfaketown and Botpocalypse Soon. No, you hang up. No, you hang up.

Overcoming Bias. Feature steering mostly preserves accuracy, up to a point.

They Took Our Jobs. Which jobs will they be allowed to take?

The Art of the Jailbreak. Don’t jailbreak, just steer us through mythic space.

Get Involved. Apollo Research position on the EU AI Act.

Introducing. Perplexity features, Sonnet in GitHub Copilot, prompting essentials.

In Other AI News. We have the chips, also the voice modes.

Quiet Speculations. AI is going to be a huge deal.

Thanks for the Memos: Introduction and Competitiveness. Wh

...
The Compendium, A full argument about extinction risk from AGI
by adamShimi — 2024-10-31T12:01:51.714Z
Link: https://www.thecompendium.ai/

We (Connor Leahy, Gabriel Alfour, Chris Scammell, Andrea Miotti, Adam Shimi) have just published

The Compendium

, which brings together in a single place the most important arguments that drive our models of the AGI race, and what we need to do to avoid catastrophe.

We felt that something like this has been missing from the AI conversation. Most of these points have been shared before, but a “comprehensive worldview” doc has been missing. We’ve tried our best to fill this gap, and welcome feedback and debate about the arguments. The Compendium is a living document, and we’ll keep updating it as we learn more and change our minds.

We would appreciate your feedback, whether or not you agree with us:

If you do agree with us, please point out where you think the arguments can be made stronger, and contact us if there are ways you’d be interested in collaborating in the future.
If you disagree with us, please let us know where our argument loses you and which points are the most significant cruxes - we welcome debate.

Here is the twitter thread and the summary:

The Compendium aims to present a coherent worldview about the extinction risks of artificial general intelligence (AGI), an artificial intelligence that exceeds that of humans, in a way that is accessible to non-technical readers who have no prior knowledge of AI. A reader should come away with an understanding of the current landscape, the race to AGI, and its existential stakes.

AI progress is rapidly converging on building AGI, driven by a brute-force paradigm that is bottlenecked by resources, not insights. Well-resourced, ideologically motivated individuals are driving a corporate race to AGI. They are now backed by Big Tech, and will soon have the support of nations.

People debate whether or not it is possible to build AGI, but most of the discourse is rooted in pseudoscience. Because humanity lacks a formal theory of intelligence, we must operate by the empirical observation that AI capabilities are increasing rapidly, surpassing human benchmarks at an unprecedented pace.

As more and more human tasks are automated, the gap between artificial and human intelligence shrinks. At the point when AI is able to do all of the tasks a human can on a computer, it will functionally be AGI and able to conduct the same AI research that we can. Should this happen, AGI will quickly scale to superintelligenc

...
Some Preliminary Notes on the Promise of a Wisdom Explosion
by Chris_Leong — 2024-10-31T09:21:11.623Z
Link: https://aiimpacts.org/some-preliminary-notes-on-the-promise-of-a-wisdom-explosion/

This post is one-half of my third-prize winning entry for the AI Impacts Essay Competition on the Automation of Wisdom and Philosophy. It proposes a "wisdom explosion" as an alternative to an intelligence explosion that is safer from the standpoint of differential technological development.
What TMS is like
by Sable — 2024-10-31T00:44:22.612Z
Link: https://affablyevil.substack.com/p/what-tms-is-like

There are two nuclear options for treating depression: Ketamine and TMS; This post is about the latter.

TMS stands for Transcranial Magnetic Stimulation. Basically, it fixes depression via magnets, which is about the second or third most magical things that magnets can do.

I don’t know a whole lot about the neuroscience - this post isn’t about the how or the why. It’s from the perspective of a patient, and it’s about the what.

What is it like to get TMS?

TMS

The Gatekeeping

For Reasons™, doctors like to gatekeep access to treatments, and TMS is no different. To be eligible, you generally have to have tried multiple antidepressants for several years and had them not work or stop working. Keep in mind that, while safe, most antidepressants involve altering your brain chemistry and do have side effects.

Since TMS is non-invasive, doesn’t involve any drugs, and has basically little to no risk or side effects, doctors want to make sure that you’ve tried everything else first.

(I suppose if the American healthcare system made sense, a lot of people would be out of the job.)

In any case, when I applied for the treatment (after talking to my psychiatrist about it), I went to a clinic that does the treatment, and in the first appointment went through the standard rigmarole with insurance information and medical history. In order to be eligible, I had to complete a form indicating my history with antidepressants (I’ve tried several), and some other surveys to indicate how depressed I was (very).

After being informed I was a good candidate for the treatment, I scheduled an initial session with a doctor, which would include a motor threshold test.

Motor Threshold Test

Given that the entire treatment revolves around stimulating one’s brain with a magnetic field, the question comes up: how do they know what region of the brain to stimulate, and where it is in your skull? Also, how do they know how powerful to make the magnetic field?

The answer does not involve an MRI, even though that is also a medical procedure involving magnets that go click.

Instead, they do something called a Motor Threshold Test. Basically, there’s a band of the brain responsible for motor function:

The magnetic coil in the TMS machine is placed next to your head along this motor function area:

And then they activate the machine, zapping your brain. Because specific areas of this band are responsible fo...
AI Safety at the Frontier: Paper Highlights, October '24
by gasteigerjo — 2024-10-31T00:09:33.522Z
Link: https://aisafetyfrontier.substack.com/p/paper-highlights-october-24

This is a selection of AI safety paper highlights in October 2024, from my blog "AI Safety at the Frontier". The selection primarily covers ML-oriented research. It's only concerned with papers (arXiv, conferences etc.), not LessWrong or Alignment Forum posts. As such, it should be a nice addition for people primarily following the forum, who might otherwise miss outside research.

tl;dr

Paper of the month:

Anthropic's sabotage evaluations test whether frontier models can undermine safety measures while avoiding detection. Results on Claude 3 suggest current models don't yet cross concerning thresholds, but highlight the importance of realistic oversight conditions in evaluations.

Research highlights:

New benchmarks for testing harmful AI agent capabilities show concerning gaps in safety measures when models use tools or interact with malicious users.
LLMs are better at predicting their own behavior than other models.
Progress on estimating probabilities of rare model outputs.
Efficient LLM-based steganography suggests high limits for what LLMs could surreptitiously encode.
An investigation of the residual error of SAEs suggests unexplained patterns are mostly linear.
Formal contracts in multi-agent RL could help align incentives between AI systems.

Paper of the month

Sabotage Evaluations for Frontier Models

Read the paper

[Anthropic, Redwood, CEIP]

Suspiciousness scores of control non-attacks, low-impact attacks, and verified high-impact attacks. Verified attacks have higher suspiciousness.

As AI systems become more capable, there is growing concern about misaligned systems that might actively scheme to achieve harmful goals. While much attention has focused on directly dangerous capabilities like generating cyberattacks or bioweapons, a more subtle but equally critical risk comes from systems that could subvert our safety measures themselves.

Such systems would need to appear safe during evaluation while retaining dangerous capabilities, manipulate human decisions without raising suspicion, and potentially undermine safety measures from within AI development organizations. These "sabotage capabilities" could pose an existential risk, as they might allow dangerous AI systems to be deployed without adequate safeguards or cause catastrophic failures in high-stakes situations. Our paper of the month introduces a framework for evaluating such capabilities and tests whether curr...
Standard SAEs Might Be Incoherent: A Choosing Problem & A “Concise” Solution
by Kola Ayonrinde — 2024-10-30T22:50:45.642Z

This work was produced as part of the ML Alignment & Theory Scholars Program - Summer 24 Cohort.

TL;DR

The current approach to explaining model internals is to (1) disentangle the language model representations into feature activations using a Sparse Autoencoder (SAE) and then (2) explain the corresponding feature activations using an AutoInterp process. Surprisingly, when formulating this two stage problem, we find that the problem is not well defined for standard SAEs: an SAE's feature dictionary can provide multiple different explanations for a given set of neural activations, despite both explanations having equivalent sparsity (L0) and reconstruction (MSE) loss. We denote this non-uniqueness, the incoherence property of SAE. We illustrate this incoherence property in practise with an example from Gemma Scope SAEs. We then suggest a possible well-defined version of the problem using the Minimum Description Length SAE (MDL-SAE) approach, introduced here.

Structure

We begin in Section 1 by outlining how SAE-style feature-based explanation is done.
In Section 2, we identify the incoherence problem in the current two-step approach to interpreting neural activations and show that there is no simple solution to this problem within the standard SAE setting.
In Section 3, we suggest a possible solution in terms of MDL-SAEs.
We close with a discussion and some meta-reflections on MechInterp research in sections 4 and 5.

1. The Setup

In interpretability, we often have some neural activations that we would like for humans to understand in terms of natural language statements[1].

Unfortunately, the neurons themselves are not very amenable to interpretation due to superposition and the phenomena of polysemantic neurons. It is now customary to use sparse autoencoders (SAEs) in order to aid in the interpretation process by working in some more disentangled feature space rather than directly in the neuron space. We will denote this the Feature Approach to explaining neural activations - explaining neural activations via the feature space.

We can contrast the Feature Approach with previous work like Bills et al. (2023) and Olah et al. (2020) who focused on interpreting neurons directly. In the Feature Approach to explanations however, we explain neuron activations by explaining the features which correspond to those neuron activations. Lee Sharkey refers to these two steps as t...
Generic advice caveats
by Saul Munn — 2024-10-30T21:03:07.185Z
Link: https://www.brasstacks.blog/caveats/

You were (probably) linked here from some advice. Unfortunately, that advice has some caveats. See below:

There exist some people who should not do the advice.
Moreover,

people are different

.
More moreover, situations are different. What worked there/then might not work here/now.
Some of the advice is missing context, contradictory, inaccurate, misleading, won’t

replicate

, or is downright false or useless.

Consider reversing the advice.

The author’s incentives might be misaligned with your goals. The author might say something like “these are affiliate links” or “I’m investing in this company,” but sometimes they just get more clicks/upvotes/karma/likes/whatever from overconfidence or exaggeration. Or they might just get a kick out of giving advice.
You might not be the target audience for the advice.
The author of the advice might be explaining the advice poorly, even if it’s a good idea. (Corellary: the author might be explaining the advice really convincingly, but it still might not be good advice.)
Reading the advice selects for advice that you’re liable to read; hearing the advice selects for advice that you’re liable to hear; etc. You’ll read a lot more blog posts telling you to start a blog than ones telling you not to, because the people who have good takes on why you shouldn’t write a blog aren’t writing a blog posts about it.
The advice might not help at all. It might even make your problem worse. Even if the advice helps, it might not totally solve your problem, or it might create new problems you didn’t expect.
The advice is for entertainment value only, and not professional advice. The author is not a [lawyer, doctor, whatever], and if they are, they’re not acting in a professional capacity.
Following the advice might relevantly change your frame/outlook/goals/etc in life. This could, for instance, cause your success criteria to change such that it’s impossible for the advice to ever “succeed” by your current self’s version of “success” — only by past versions.
The advice might assume certain capabilities, knowledge, connections, or other resources that you don’t have.
The advice might have second, third, or nth order effects. Even the sign of these can be extremely difficult to predict, let alone their general shape, let alone specifics.
Stolen from

dynomight's essay on advice

, and bastardized to fit the context of this list:

The advice might be incomplete wit

...
AI as a powerful meme, via CGP Grey
by TheManxLoiner — 2024-10-30T18:31:58.544Z

In episode 158 of the Cortex podcast, CGP Grey gives their high-level reason why they are worried about AI.

My one line summary: AI should not be compared to nuclear weapons but instead to biological weapons or memes, which evolve under the implicit evolutionary pressures that exist, leading to AI's that are good at surviving and replicating.

The perspective is likely known by many in the community already, but I had not heard it before. Interestingly, there have actually been experiments where they just put random strings of code in an environment where they interact, and self-replicating code appeared. See Cognitive Revolution podcast on 'Computational Life: How Self-Replicators Arise from Randomness', with Google researchers Ettore Randazzo and Luca Versari.

I quote the relevant part of the podcast below, but I recommend listening because the emotion and delivery is impactful. It is from 1:22:00 onwards.

To be explicit and not beat around the bush, when I try to think, “Oh, what is beyond this barrier, beyond which it might be impossible to predict?” it's like, well, if I’m just in Vegas and placing odds on this roulette wheel, almost all of those outcomes are extraordinarily bad for the human species. There are potentially paths where it goes well, but most are extremely bad for a whole bunch of reasons.

I think of it like this: people who are concerned like me often analogize AI to something like building nuclear weapons. It’s like, “Ah, we’re building a thing that could be really dangerous.” But I just don’t think that’s the correct comparison, because a nuclear weapon is a tool. It's a tool like a hammer. It’s a very bad hammer, but it is fundamentally mechanical in a particular way.

But the real difference, where do I disagree with people, where do other people disagree with me, is that I think a much more correct way to think about AI is to compare it to biological weaponry. You’re building a thing able to act in the world differently than how you constructed it. That’s what biological weapons are—they’re alive. A nuclear bomb doesn’t accidentally leave the factory on its own, whereas biological weapons do, can, and have. And once a biological weapon is out in the world, it can develop in ways that you never anticipated ahead of time.

That’s the way I think about these AI systems.
[...]

The reason I like to talk about it this way, particularly with bio

...
How might language influence how an AI "thinks"? 
by bodry — 2024-10-30T17:41:04.460Z

In some fiction I've encountered humans are treated as being linguistically deterministic. For example, in Arrival humans are strongly linguistically deterministic and a human that learns the alien language is able to escape the linear ordering of time that English imposes. In 1984 Newspeak is a language created for the purpose of limiting human expression particularly those of political will.

Is there any evidence that the language a model is trained in significantly effects any abilities, like that of deception? Or does the language a model is trained on not matter at all?

What is the near-future feasibility of training an AI on a language with a limited amount of publicly available data?
Motivation control
by Joe Carlsmith — 2024-10-30T17:15:50.881Z

(This is the second in a series of four posts about how we might solve the alignment problem. See the first post for an introduction to this project and a summary of the posts that have been released thus far.)

Summary

In my last post, I laid out the ontology I’m going to use for thinking about approaches to solving the alignment problem (that is, the problem of building superintelligent AI agents, and becoming able to elicit their beneficial capabilities, with succumbing to the bad kind of AI takeover[1]). In particular:

I distinguished between option control (trying to ensure that an AI’s options have specific properties) and motivation control (trying to ensure that an AI’s motivations have specific properties);
I suggested that our approach to the alignment problem may well need to combine both;
I offered a framework for thinking about what this looks like (i.e., “incentive structure safety cases”);
I described the space of high-level possibilities for option/motivation control in a bit more detail – in particular, distinguishing between what I call “internal” vs. “external” variables (roughly: the former are what a “black box” AI hides), and between “inspection” vs. “intervention” directed at a given variable.
I discussed some of the dynamics surrounding what I call “AI-assisted improvements.”

In this post, I’m going to offer a more detailed analysis of the available approaches to motivation control in particular. Here’s a summary:

I start by describing what I see as the four key issues that could make motivation control difficult. These are:

Generalization with no room for mistakes: you can’t safely test on the scenarios you actually care about (i.e., ones where the AI has a genuine takeover option), so your approach needs to generalize well to such scenarios on the first critical try (and the second, the third, etc).
Opacity: if you could directly inspect an AI’s motivations (or its cognition more generally), this would help a lot. But you can’t do this with current ML models.
Feedback quality: to the extent you are relying on behavioral feedback in shaping your AI’s motivations, flaws in a feedback signal could create undesirable results, and avoiding such flaws might require superhuman evaluation ability. (Also you need to train on a sufficient range of cases.)
Adversarial dynamics: the AI whose motivations you’re trying to control

...