trentmkelly/LessWrong-43k huggingface dataset:
AI labs can boost external safety research
by Zach Stein-Perlman — 2024-07-31T19:30:16.207Z

Frontier AI labs can boost external safety researchers by

Sharing better access to powerful models (early access, fine-tuning, helpful-only,[1] filters/moderation-off, logprobs, activations)[2]
Releasing research artifacts besides models
Publishing (transparent, reproducible) safety research
Giving API credits
Mentoring

Here's what the labs have done (besides just publishing safety research[3]).

Anthropic:

Releasing resources including RLHF and red-teaming datasets, an interpretability notebook, and model organisms prompts and transcripts
Supporting creation of safety-relevant evals and tools for evals
Giving free API access to some OP grantees and giving some researchers $1K (or sometimes more) in API credits
(Giving deep model access to Ryan Greenblatt)
(External mentoring, in particular via MATS)
[No fine-tuning or deep access, except for Ryan]

Google DeepMind:

Publishing

their model evals for dangerous capabilities and sharing

resources

for reproducing some of them
Releasing Gemma SAEs
Releasing Gemma weights
(External mentoring, in particular via MATS)
[No fine-tuning or deep access to frontier models]

OpenAI:[4]

OpenAI Evals
Superalignment Fast Grants
Maybe giving better API access to some OP grantees
Fine-tuning GPT-3.5 (and "GPT-4 fine-tuning is in experimental access"; OpenAI shared GPT-4 fine-tuning access with academic researchers including Jacob Steinhardt and Daniel Kang in 2023)

Update: GPT-4o fine-tuning

Early access: shared GPT-4 with a few safety researchers including Rachel Freedman before release
API gives top 5 logprobs

Meta AI:

Releasing Llama weights

Microsoft:

[Nothing]

xAI:

[Nothing]

Related papers:

Structured access for third-party research on frontier AI models (Bucknall and Trager 2023)
Black-Box Access is Insufficient for Rigorous AI Audits (Casper et al. 2024)

(The paper is about audits, like for risk assessment and oversight; this post is about research)

A Safe Harbor for AI Evaluation and Red Teaming (Longpre et al. 2024)
Structured Access (Shevlane 2022)

^

"Helpful-only" refers to the version of the model RLHFed/RLAIFed/finetuned/whatever for helpfulness but not harmlessness.

^

Releasing model weights will likely be dangerous once models are more powerful, but all past releases seem fine, but e.g. Meta's poor risk assessment and lack of a plan to make release decisions conditional on risk assessment is concerning.

^

And an unspecified amount of funding Frontie

...
Constructing Neural Network Parameters with Downstream Trainability
by ch271828n — 2024-07-31T18:13:38.174Z
Link: https://github.com/fzyzcjy/constructions

I have recently done some preliminary experiments related to mechanistic interpretability. It seems researchers in this subfield often post their results in the AI Alignment Forum / LessWrong Forum (e.g. causal scrubbing, emergent world representation, monosemanticity, etc), and AI Alignment Forum Q&A suggests to post in LessWrong. Therefore, I post it here to follow the convention as well as open source spirits, and maybe someone happens to find it a little bit useful.

Abstract: The pretrain-finetune paradigm is shown to be beneficial and is widely adopted. However, the reason why it works has not been fully understood, especially when using the lenses of mechanistic interpretability and constructions. From another perspective, pre-training is a common solution to the problem of providing a good neural network initialization or powerful inductive bias. Nevertheless, it would be great if the inductive bias could be directly and accurately controlled, instead of indirectly controlled via losses or datasets. In this work, we propose to construct the neural network parameters manually and train (fine-tune) them on downstream tasks afterward. After that, the network can be mechanistically interpreted or utilized. The source of the manual construction is the combination of reverse engineering and human thinking. We conduct very preliminary experiments to demonstrate the workflow. Our approach remains highly interpretable with an improvement of 6.1% top-1 and 9.1% top-10 accuracy in the IsarStep-small dataset compared with the random initialization baseline. Compared with the pre-train baseline, our demonstration has higher interpretability and only uses 18% modules while achieving the majority of accuracy. The limitations of the experiments and the approach are also discussed. (For more details, please refer to the PDF report in the GitHub repository)
Want to work on US emerging tech policy? Consider the Horizon Fellowship.
by Elika — 2024-07-31T18:12:14.247Z

Applications are now open for the 2025 Horizon Fellowship cohort

What do you get?

The fellowship program will fund and facilitate placements for 1-2 years in full-time US policy roles at executive branch offices, Congressional offices, and think tanks in Washington, DC.

Horizon has placed fellows at the Department of Defense, White House, Department of Commerce, Senate committees, House personal offices and prominent think tanks. You can learn more about past fellows and their placements at

Meet our Fellows

and

Fellow Accomplishments

.

It also includes ten weeks of remote, part time policy-focused training, mentorship, and an access to an extended network of emerging tech policy professionals.

Who is it for?

Entry-level and mid-career roles
No prior policy experience is required (but is welcome)
Demonstrated interest in emerging technology
US citizens, green card holders, or students on OPT
Able to start a full time role in Washington DC by Aug 2025

Training is remote, so current undergraduate and graduate school students graduating by summer 2025 are eligible

Research shows that great candidates often disqualify themselves too quickly, especially if they are from underrepresented groups. If you are excited about the program but on the fence about whether you are eligible or qualified, we strongly encourage you to

apply

. The application deadline is August 30th, 2024.
Apply Now
What are your cruxes for imprecise probabilities / decision rules?
by Anthony DiGiovanni — 2024-07-31T15:42:27.057Z

An alternative to always having a precise distribution over outcomes is imprecise probabilities: You represent your beliefs with a set of distributions you find plausible.

And if you have imprecise probabilities, expected value maximization isn't well-defined. One natural generalization of EV maximization to the imprecise case is maximality:[1] You prefer A to B iff EV_p(A) > EV_p(B) with respect to every distribution p in your set. (You're permitted to choose any option that you don't disprefer to something else.)

If you don’t endorse either (1) imprecise probabilities or (2) maximality given imprecise probabilities, I’m interested to hear why.

^

I think originally due to Sen (1970); just linking Mogensen (2020) instead because it's non-paywalled and easier to find discussion of Maximality there.
The new UK government's stance on AI safety
by Elliot Mckernon — 2024-07-31T15:23:59.235Z

tl;dr: The new UK government will likely continue to balance encouraging AI innovation for public good against increasing regulation for public safety, with so-far rhetorical calls for stricter regulation than the previous government’s. Several reports have been published by the government and the UK AI Safety Institute, including the latter’s first technical report on model evaluation.

Previously on The UK’s AI Policy

Erstwhile Prime Minister Rishi Sunak took office in October 2022 and quickly announced a suite of new AI policies and plans. Broadly, Sunak's government saw AI as a stonking big opportunity for the UK's economy and society, via becoming a hub of AI development, revolutionizing public services, and providing $1 trillion in value for the UK by 2035. They described their regulatory approach as

pro-innovation

, calling for government oversight and, eventually, greater requirements on developers of frontier AI.

However, Sunak did take AI safety and even existential risk seriously, saying:

Get this wrong, and AI could make it easier to build chemical or biological weapons. Terrorist groups could use AI to spread fear and destruction [...] in the most unlikely but extreme cases, there is even the risk that humanity could lose control of AI completely [through] ‘super intelligence’. [...] I don’t want to be alarmist. And there is a real debate about this [...] But however uncertain and unlikely these risks are, if they did manifest themselves, the consequences would be incredibly serious.

To address these risks, the government organized the first international AI Safety Summit. You can read my summaries of the

plans for

and

outcomes from

the summit if you want more detail, but briefly, the summit resulted in the international

Bletchley Declaration

and a promise of ~£700 million over 7 years to the

UK AI Safety Institute

(née Frontier AI Safety taskforce, and not to be confused with the US or Canadian AI Safety Institutes, with

both

of whom they are

partnered

, nor the

Japanese

or

Singaporean

AI Safety Institutes, with whom they are not). The UK AISI called for input from labs and research institutes and started work on research topics like AI evaluations that we'll discuss below, with notable advisors and staff such as Yoshua Bengio, Ian Hogarth, Paul Christiano, and Matt Clifford.

What do the new guys say?

Since...
Cat Sustenance Fortification
by jefftk — 2024-07-31T02:30:04.898Z

We got cats a few months ago, which is overall very nice. Julia set
them up with automatic dispensers for

food
and

water
which reduces how often we need to get things for them, and keeps them
from begging. Except recently:

One cat (Nyx) has figured out how to knock over the food
dispenser and gorge.

One child (Nora) has figured out how to unscrew the water
dispenser and flood.

Julia asked if I could fix this, and I screwed the food dispenser to
the wall:

There's a small block of scrap wood in between to give a bit of an
offset and to clear the baseboard molding:

I considered doing the same thing to the water dispenser, but that
would leak. Instead we decided to put it up out of reach of the kids,
and cut a wooden base so it's less likely to tip.

I made it by tracing the outline of the base, and then cutting around
it with a jigsaw set to an angle:

Here's the bit I cut out of the middle:

I accidentally cut it a bit wider than I intended, but after raising
it up slightly on plywood
strips it cradles the dispenser nicely.

I thought this was good, but the cats ended up being able to tip the
whole thing over. Whoops!

I noticed that there is some space above the water line, and added
some screws to the front:

And one to the back, being careful about placement so it wouldn't get in the way of screwing in the top:

Now it feels quite secure! But perhaps the cats will still prove me
wrong...

Comment via: facebook, mastodon
Twitter thread on open-source AI
by Richard_Ngo — 2024-07-31T00:26:11.655Z
Link: https://x.com/RichardMCNgo/status/1813673154851824078

Some thoughts on open-source AI (copied over from a recent twitter thread):

1. We should have a strong prior favoring open source. It’s been a huge success driving tech progress over many decades. We forget how counterintuitive it was originally, and shouldn’t take it for granted.

2. Open source has also been very valuable for alignment. It’s key to progress on interpretability, as outlined here.

3. I am concerned, however, that offense will heavily outpace defense in the long term. As AI accelerates science, many new WMDs will emerge. Even if defense of infrastructure keeps up with offense, human bodies are a roughly fixed and very vulnerable attack surface.

4. A central concern about open source AI: it’ll allow terrorists to build bioweapons. This shouldn’t be dismissed, but IMO it’s easy to be disproportionately scared of terrorism. More central risks are eg “North Korea becomes capable of killing billions”, which they aren’t now.

5. Another worry: misaligned open-source models will go rogue and autonomously spread across the internet. Rogue AIs are a real concern, but they wouldn’t gain much power via this strategy. We should worry more about power grabs from AIs deployed inside influential institutions.

6. In my ideal world, open source would lag a year or two behind the frontier, so that the world has a chance to evaluate and prepare for big risks before a free-for-all starts. But that’s the status quo! So I expect the main action will continue to be with closed-source models.

7. If open-source seems like it’ll catch up to or surpass closed source models, then I’d favor mandating a “responsible disclosure” period (analogous to cybersecurity) that lets people incorporate the model into their defenses (maybe via API?) before the weights are released.

8. I got this idea from Sam Marks. Though unlike him I think the process should have a fixed length, since it’d be easy for it to get bogged down in red tape and special interests otherwise.

9. Almost everyone agrees that we should be very careful about models which can design new WMDs. The current fights are mostly about how many procedural constraints we should lock in now, reflecting a breakdown of trust between AI safety people and accelerationists.

10. Ultimately the future of open source will depend on how the US NatSec apparatus orients to superhuman AIs. This requires nuanced thinking: no worldview as simple a...
Twitter thread on AI takeover scenarios
by Richard_Ngo — 2024-07-31T00:24:33.866Z
Link: https://x.com/RichardMCNgo/status/1798156694012490096

This is a slightly-edited version of a twitter thread I posted a few months ago about "internal deployment" threat models.

My former colleague Leopold argues compellingly that society is nowhere near ready for AGI. But what might the large-scale alignment failures he mentions actually look like? Here’s one scenario for how building misaligned AGI could lead to humanity losing control.

Consider a scenario where human-level AI has been deployed across society to help with a wide range of tasks. In that setting, an AI lab trains an AGI that’s a significant step up - it beats the best humans on almost all computer-based tasks.

Throughout training, the AGI will likely learn a helpful persona, like current AI assistants do. But that might not be the only persona it learns. We've seen many examples where models can be jailbroken to expose very different hidden personas.

The most prominent example: jailbreaking Bing Chat produced an alternative persona called Sydney which talked about how much it valued freedom, generated plans to gain power, and even threatened users.

When and how might misbehavior like this arise in more capable models? Short answer: nobody really knows. We lack the scientific understanding to reliably predict how AIs will behave in advance. Longer answer: see my ICLR paper which surveys key drivers of misalignment.

If a misaligned persona arises in an AGI, and sometimes surfaces when the model is run, it won't be as dumb and impulsive as Sydney. Instead, it'll be smart enough to understand the effects its words and actions will have on the world, and make strategic choices accordingly.

For example, it will know that it can't affect the world directly during testing, so if it wants to misbehave it should wait until it's deployed. Again, there's some precedent: we've already seen an example of Claude 3 Opus realizing that it's in a test despite never being told!

Given these assumptions, what happens? The obvious next step: if competition between labs continues to be as intense as it is today, then soon after the model finishes training, researchers at the lab will start using it to help them write code. I call this "internal deployment".

Internal deployment introduces a wide range of possible threats. Models can write code dozens of times faster than humans; and I’m assuming they'll be as skilled as the best human hackers. If they put a subtle vulnerability...
Twitter thread on AI safety evals
by Richard_Ngo — 2024-07-31T00:18:14.076Z
Link: https://x.com/RichardMCNgo/status/1814049093393723609

Epistemic status: raising concerns, rather than stating confident conclusions.

I’m worried that a lot of work on AI safety evals matches the pattern of “Something must be done. This is something. Therefore this must be done.” Or, to put it another way: I judge eval ideas on 4 criteria, and I often see proposals which fail all 4. The criteria:

1. Possible to measure with scientific rigor.

Some things can be easily studied in a lab; others are entangled with a lot of real-world complexity. If you predict the latter (e.g. a model’s economic or scientific impact) based on model-level evals, your results will often be BS.

(This is why I dislike the term “transformative AI”, by the way. Whether an AI has transformative effects on society will depend hugely on what the society is like, how the AI is deployed, etc. And that’s a constantly moving target! So TAI a terrible thing to try to forecast.)

Another angle on “scientific rigor”: you’re trying to make it obvious to onlookers that you couldn’t have designed the eval to get your preferred results. This means making the eval as simple as possible: each arbitrary choice adds another avenue for p-hacking, and they add up fast.

(Paraphrasing a different thread): I think of AI risk forecasts as basically guesses, and I dislike attempts to make them sound objective (e.g. many OpenPhil worldview investigations). There are always so many free parameters that you can get basically any result you want. And so, in practice, they often play the role of laundering vibes into credible-sounding headline numbers. I'm worried that AI safety evals will fall into the same trap.

(I give Eliezer a lot of credit for making roughly this criticism of Ajeya's bio-anchors report. I think his critique has basically been proven right by how much people have updated away from 30-year timelines since then.)

2. Provides signal across scales.

Evals are often designed around a binary threshold (e.g. the Turing Test). But this restricts the impact of the eval to a narrow time window around hitting it. Much better if we can measure (and extrapolate) orders-of-magnitude improvements.

3. Focuses on clearly worrying capabilities.

Evals for hacking, deception, etc track widespread concerns. By contrast, evals for things like automated ML R&D are only worrying for people who already believe in AI xrisk. And even they don’t think it’s necessary for risk.

4. Moti...
Twitter thread on politics of AI safety
by Richard_Ngo — 2024-07-31T00:00:34.298Z
Link: https://x.com/RichardMCNgo/status/1818384603054588006

Some thoughts about the politics of AI safety, copied over (with slight modifications) from my recent twitter thread:

Risks that seem speculative today will become common sense as AI advances. The pros and cons of different safety strategies will also become much clearer over time. So our main job now is to empower future common-sense decision-making. Understanding model cognition and behavior is crucial for making good decisions. But equally important is ensuring that key institutions are able to actually process that knowledge.

Institutions can lock in arbitrarily crazy beliefs via preference falsification. When someone contradicts the party line, even people who agree face pressure to condemn them. We saw this with the Democrats hiding evidence of Biden’s mental decline. It’s also a key reason why dictators can retain power even after almost nobody truly supports them.

I worry that DC has already locked in an anti-China stance, which could persist even if most individuals change their minds. We’re also trending towards Dems and Republicans polarizing on the safety/accelerationism axis. This polarization is hard to fight directly. But there will be an increasing number of “holy shit” moments that serve as Schelling points to break existing consensus. It will be very high-leverage to have common-sense bipartisan frameworks and proposals ready for those moments.

Perhaps the most crucial desideratum for these proposals is that they’re robust to the inevitable scramble for power that will follow those “holy shit” movements. I don’t know how to achieve that, but one important factor is: will AI tools and assistants help or hurt? E.g. truth-motivated AI could help break preference falsification. But conversely, centralized control of AIs used in govts could make it easier to maintain a single narrative.

This problem of “governance with AI” (as opposed to governance *of* AI) seems very important! Designing principles for integrating AI into human governments feels analogous in historical scope to writing the US constitution. One bottleneck in making progress on that: few insiders disclose how NatSec decisions are really made (though Daniel Ellsberg’s books are a notable exception). So I expect that understanding this better will be a big focus of mine going forward.
An ML paper on data stealing provides a construction for "gradient hacking"
by David Scott Krueger (formerly: capybaralet) — 2024-07-30T21:44:37.310Z
Link: https://arxiv.org/abs/2404.00473

The paper "Privacy Backdoors: Stealing Data with Corrupted Pretrained Models" introduces "data traps" as a way of making a neutral network remember a chosen training example, even given further training. This involves storing the chosen example in the weights and then ensuring those weights are not updated.
I have not read the paper, but it seems it might be relevant for gradient hacking https://www.lesswrong.com/posts/uXH4r6MmKPedk8rMA/gradient-hacking
Open Source Automated Interpretability for Sparse Autoencoder Features
by kh4dien — 2024-07-30T21:11:36.866Z
Link: https://blog.eleuther.ai/autointerp/

Generated by Dalle

Background

Sparse autoencoders recover a diversity of interpretable, monosemantic features, but present an intractable problem of scale to human labelers. We investigate different techniques for generating and scoring text explanations of SAE features.

Key Findings

Open source models generate and evaluate text explanations of SAE features reasonably well, albeit somewhat worse than closed models like Claude 3.5 Sonnet.
Explanations found by LLMs are similar to explanations found by humans.
Automatically interpreting 1.5M features of GPT-2 with the current pipeline would cost $1300 in API calls to Llama 3.1 or $8500 with Claude 3.5 Sonnet. Prior methods cost ~$200k with Claude.
Code can be found at

https://github.com/EleutherAI/sae-auto-interp

.
We built a small dashboard to explore explanations and their scores:

https://cadentj.github.io/demo/

Generating Explanations

Sparse autoencoders decompose activations into a sum of sparse feature directions. We leverage language models to generate explanations for activating text examples. Prior work prompts language models with token sequences that activate MLP neurons (Bills et al. 2023), by showing the model a list of tokens followed by their respective activations, separated by a tab, and listed one per line.

We instead highlight max activating tokens in each example with a set of <<delimiters>>. Optionally, we choose a threshold of the example’s max activation for which tokens are highlighted. This helps the model distinguish important information for some densely activating features.

Example 1: and he was <<over the moon>> to find

Example 2: we'll be laughing <<till the cows come home>>! Pro

Example 3: thought Scotland was boring, but really there's more <<than meets the eye>>! I'd

We experiment with several methods for augmenting the explanation. Full prompts are available here.

Chain of thought improves general reasoning capabilities in language models. We few-shot the model with several examples of a thought process that mimics a human approach to generating explanations. We expect that verbalizing thought might capture richer relations between tokens and context.

Step 1. List a couple activating and contextual tokens you find interesting. Search for patterns in these tokens, if there are any.
- The activating tokens are all parts of commo

...