trentmkelly/LessWrong-43k huggingface dataset:
DeepMind: Frontier Safety Framework
by Zach Stein-Perlman — 2024-05-17T17:30:02.504Z
Link: https://deepmind.google/discover/blog/introducing-the-frontier-safety-framework/

DeepMind's RSP is here: blogpost, full document. Compare to Anthropic's RSP, OpenAI's RSP ("PF"), and METR's Key Components of an RSP.

(Maybe it doesn't deserve to be called an RSP — it doesn't contain commitments, it doesn't really discuss safety practices as a function of risk assessment results, and the deployment safety practices it mentions are kinda vague and only about misuse.)

Edit: new blogpost with my takes. Or just read DeepMind's doc; it's really short.

Hopefully DeepMind was rushing to get something out before the AI Seoul Summit next week and they'll share stronger and more detailed stuff soon. If this is all we get for months, it's quite disappointing.

Excerpt

Today, we are introducing our

Frontier Safety Framework

- a set of protocols for proactively identifying future AI capabilities that could cause severe harm and putting in place mechanisms to detect and mitigate them. Our Framework focuses on severe risks resulting from powerful capabilities at the model level, such as exceptional agency or sophisticated cyber capabilities. It is designed to complement our alignment research, which trains models to act in accordance with human values and societal goals, and Google’s existing suite of AI responsibility and safety

practices

.

The Framework is exploratory and we expect it to evolve significantly as we learn from its implementation, deepen our understanding of AI risks and evaluations, and collaborate with industry, academia, and government. Even though these risks are beyond the reach of present-day models, we hope that implementing and improving the Framework will help us prepare to address them. We aim to have this initial framework fully implemented by early 2025.

The Framework

The first version of the Framework announced today builds on our

research

on

evaluating

critical capabilities in frontier models, and follows the emerging approach of

Responsible Capability Scaling.

The Framework has three key components:

Identifying capabilities a model may have with potential for severe harm. To do this, we research the paths through which a model could cause severe harm in high-risk domains, and then determine the minimal level of capabilities a model must have to play a role in causing such harm. We call these “Critical Capability Levels” (CCLs), and they guide our evaluation and mitigation approach.
Evaluating our frontier models periodically to d

...
Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning
by Dan Braun — 2024-05-17T16:25:02.267Z
Link: https://arxiv.org/abs/2405.12241

A short summary of the paper is presented below.

This work was produced by Apollo Research in collaboration with Jordan Taylor (MATS + University of Queensland) .

TL;DR: We propose end-to-end (e2e) sparse dictionary learning, a method for training SAEs that ensures the features learned are functionally important by minimizing the KL divergence between the output distributions of the original model and the model with SAE activations inserted. Compared to standard SAEs, e2e SAEs offer a Pareto improvement: They explain more network performance, require fewer total features, and require fewer simultaneously active features per datapoint, all with no cost to interpretability. We explore geometric and qualitative differences between e2e SAE features and standard SAE features.

Introduction

Current SAEs focus on the wrong goal: They are trained to minimize mean squared reconstruction error (MSE) of activations (in addition to minimizing their sparsity penalty). The issue is that the importance of a feature as measured by its effect on MSE may not strongly correlate with how important the feature is for explaining the network's performance.

This would not be a problem if the network's activations used a small, finite set of ground truth features -- the SAE would simply identify those features, and thus optimizing MSE would have led the SAE to learn the functionally important features. In practice, however, Bricken et al. observed the phenomenon of feature splitting, where increasing dictionary size while increasing sparsity allows SAEs to split a feature into multiple, more specific features, representing smaller and smaller portions of the dataset. In the limit of large dictionary size, it would be possible to represent each individual datapoint as its own dictionary element.

Since minimizing MSE does not explicitly prioritize learning features based on how important they are for explaining the network's performance, an SAE may waste much of its fixed capacity on learning less important features. This is perhaps responsible for the observation that, when measuring the causal effects of some features on network performance, a significant amount is mediated by the reconstruction residual errors (i.e. everything not explained by the SAE) and not mediated by SAE features (Marks et al.).

Given these issues, it is therefore natural to ask how we can identify the functionally im...
AISafety.com – Resources for AI Safety
by Søren Elverlin — 2024-05-17T15:57:11.712Z

There are many resources for those who wish to contribute to AI Safety, such as courses, communities, projects, jobs, events and training programs, funders and organizations. However, we often hear from people that they have trouble finding the right resources. To address this, we've built AISafety.com as a central hub—a list-of-lists—where community members maintain and curate these resources to increase their visibility and accessibility.

In addition to presenting resources, the website is optimized to be an entry point for newcomers to AI Safety, capable of funneling people towards understanding and contributing.

The website was developed on a shoestring budget, relying extensively on volunteers and Søren paying out of pocket. We do not accept donations, but if you think this is valuable, you’re welcome to help out by reporting issues or making suggestions in our tracker, commenting here, or volunteering your time to improve the site.

Feedback

If you’re up for giving us some quick feedback, we’d be keen to hear your responses to these questions in a comment:

What's the % likelihood that you will use AISafety.com within the next 1 year? (Please be brutally honest)

What list of resources will you use?
What could be changed (features, content, design, whatever) to increase that chance?

What's the % likelihood that you will send AISafety.com to someone within the next 1 year?

What could be changed (features, content, design, whatever) to increase that chance?

Any other general feedback you'd like to share

Credits

Project owner and funder – Søren Elverlin
Designer and frontend dev – Melissa Samworth
QA and resources – Bryce Robertson
Backend dev lead – nemo
Volunteers – plex, Siao Si Looi, Mathilde da Rui, Coby Joseph, Bart Jaworski, Rika Warton, Juliette Culver, Jakub Bares, Jordan Pieters, Chris Cooper, Sophia Moss, Haiku, agucova, Joe/Genarment, Kim Holder (Moonwards), de_g0od, entity, Eschaton
Reading guide embedded from AISafety.info by Aprillion (Peter Hozák)
Jobs pulled from 80,000 Hours Jobs Board and intro video adapted from 80,000 Hours’ intro with permission
Communities list, The Map of Existential Safety, AI Ecosystem Projects, Events & Training programs adapted from their respective Alignment Ecosystem Development projects (join the Discord for discussion and other projects!). Funding list adapted from Futu

...
Is There Really a Child Penalty in the Long Run?
by Maxwell Tabarrok — 2024-05-17T11:56:22.892Z
Link: https://www.maximum-progress.com/p/is-there-really-a-child-penalty-in

A couple of weeks ago three European economists published

this paper

studying the female income penalty after childbirth. The surprising headline result: there is no penalty.

Setting and Methodology

The paper uses Danish data that tracks IVF treatments as well as a bunch of demographic factors and economic outcomes over 25 years. Lundborg et al identify the causal effect of childbirth on female income using the success or failure of the first attempt at IVF as an instrument for fertility.

What does that mean? We can’t just compare women with children to those without them because having children is a choice that’s correlated with all of the outcomes we care about. So sorting out two groups of women based on observed fertility will also sort them based on income and education and marital status etc.

Successfully implanting embryos on the first try in IVF is probably not very correlated with these outcomes. Overall success is, because rich women may have the resources and time to try multiple times, for example, but success on the first try is pretty random. And success on the first try is highly correlated with fertility.

So, if we sort two groups of women based on success on the first try in IVF, we’ll get two groups that differ a lot in fertility, but aren’t selected for on any other traits. Therefore, we can attribute any differences between the groups to their difference in fertility and not any other selection forces.

Results

How do these two groups of women differ?

First of all, women who are successful on the first try with IVF are persistently more likely to have children. This random event causing a large and persistent fertility difference is essential for identifying the causal effect of childbirth.

This graph is plotting the regression coefficients on a series of binary variables which track whether a woman had a successful first-time IVF treatment X years ago. When the IVF treatment is in the future (i.e X is negative), whether or not the woman will have a successful first-time IVF treatment has no bearing on fertility since fertility is always zero; these are all first time mothers.

When the IVF treatment was one year in the past (X = 1), women with a successful first-time treatment are about 80% more likely to have a child that year than women with an unsuccessful first time treatment. This first year coefficient isn’t 1 because some women who fail their...
My Hammer Time Final Exam
by adios — 2024-05-17T09:28:13.572Z

Epistemic Status: I thought about and wrote each paragraph in 10 minutes total, with slight editing afterwards.

I hope I'm not too late to the party!

I wrote this up quite a few months ago and found that I delayed indefinitely editing it before publication. I decided it's probably best to post a not-maximally-edited version of my final exam.

This is my first post on LessWrong, and any remarks, notes, requests for revisions, and so on will be welcome!

Cognitive defect: calendar view leaky abstraction

When using a calendar or task management app, one can often judge quickly the length of a task by it's physical size on the screen. This is a useful feature of the calendar view.
This heuristic is however wildly wrong for small and short tasks. Since the tasks need to be readable, the app often displays them as taking somewhere between 5 to 15 minutes, depending on the font size and so on.
Having a large set of small 2-10 minutes task on the screen may thus feel (incorrectly) overwhelming.
The size of a task on the screen is a leaky abstraction (of it's length in time).

Rationality skill: course correction

The rational skill I'd like to talk about is course correction.
Course correction helps install replacement habits to unoptimized habits.
For example, consider the habit of not using the correct fingers when touch typing.
For that habit, a course correction might look like that: notice that you are not using the correct finger, remember that you'd like to install the habit of using the correct finger when touch typing, delete the word or letter and (while being aware to perform correctly) redo the action.
This will be useful in two regards:
1. You will actively perform the desired action many times, getting used to it
2. From the desire to optimize, your brain, not wanting to waste time on doing and undoing and action will get used to only performing the correct version.

Before you may apply course correction as a tool you need to first strongly identify with the desire to optimize. Further you need to understand how to identify the mistaken habit in real time. I find that it usually help to identify ahead of time what sort of activities may lead to that sort of mistake.

For the example of touch typing, you need to be aware of
1. Which is the correct finger for each letter, and to have an instinctual feeling of an uncomfortabl...
D&D.Sci (Easy Mode): On The Construction Of Impossible Structures
by abstractapplic — 2024-05-17T00:25:42.950Z

This is a D&D.Sci scenario: a puzzle where players are given a dataset to analyze and an objective to pursue using information from that dataset.

Duke Arado’s obsession with physics-defying architecture has caused him to run into a small problem. His problem is not – he affirms – that his interest has in any way waned: the menagerie of fantastical buildings which dot his territories attest to this, and he treasures each new time-bending tower or non-Euclidean mansion as much as the first. Nor – he assuages – is it that he’s having trouble finding talent: while it’s true that no individual has ever managed to design more than one impossible structure, it’s also true that he scarcely goes a week without some architect arriving at his door, haunted by alien visions, begging for the resources to bring them into reality. And finally – he attests – his problem is definitely not that “his mad fixation on lunatic constructions is driving him to the brink of financial ruin”, as the townsfolk keep saying: he’ll have you know he’s recently brought an accountant in to look over his expenditures, and he’s confirmed he has the funds to keep pursuing this hobby long into his old age.

Rather, his problem is the local zoning board. Concerned citizens have come together to force him to limit new creations near populated areas, claiming they “disrupt the neighbourhood character” and “conjure eldritch music to lure our children away while we sleep”. While in previous years he was free to – and did – support any qualified architect who showed up with sufficiently strange blueprints, the Duke is now forced to be selective: at present, he has fourteen applicants waiting on his word, and only four viable building sites. He finds this particularly galling, since about half the time when an architect finishes their work, the resulting building ends up not distorting the fabric of spacetime, and instead just kind of looking weird. It’s entirely possible that if he picks at random, he’ll end up with no new impossible structures at all this month, which – he asserts – would utterly break his heart.

This is where you come in. Using his records from previous years, he wants you to evaluate his current crop of architects’ plans and pick out the four most likely to successfully defy the laws of Nature. (If there are any ties, he’d like you to resolve them in favour of whichever option...
To an LLM, everything looks like a logic puzzle
by Jesse Richardson — 2024-05-16T22:21:10.512Z

I keep seeing this meme doing the rounds where people present ChatGPT with a common logic problem or riddle, only with some key component changed to make it trivial. ChatGPT has seen the original version a million times, so it gives the answer to the original, not the actually correct and obvious answer.

The idea is to show that ChatGPT isn't intelligent, it's just reciting what it's seen before in similar contexts to the one it's seeing now and there's no actual reasoning taking place. My issue with this is it's pretty clear to me that most humans fail in very similar ways, just at a slightly higher level of complexity.

The clearest way this manifests is in the adage 'to a man with a hammer, everything looks like a nail'. When presented with a new problem that bears a strong resemblance to problems we've solved before, it's a very natural response to employ the same method we used previously.

At the age of 16, I had very little understanding of statistics or machine learning but I knew about linear regression. I LOVED linear regression. If you came to me with some dataset and asked me what methodology I would use to analyze it, make predictions, anything like that, I would probably say linear regression. The dataset could be wildly, embarrassingly unsuitable for linear regression, but I would still probably suggest it as my first port of call. I certainly hope this doesn't mean that 16-year-old me was incapable of human-level reasoning.

Something I see a lot is people with monocausal narratives for history. They probably read a book about how a number of events were all caused by X. This book was probably correct! But they took it way too far and the next time someone asked them what caused some totally unrelated event, their go-to response might be X or some variation on X. In a lot of cases, this answer might be totally nonsensical, just like ChatGPT's responses above. Nevertheless this person is still able to reason intelligently, they're just performing a bunch of shortcuts in their head when deciding how to answer questions; most of the time these shortcuts work and sometimes they don't.

If someone made me do ten logic puzzles a day for a year, all with pretty similar formats and mechanisms and then one day they switched out a real logic puzzle for a fake trivial one, I'd probably get it wrong too. Long story short, I think the prompts above are...
AI Safety Institute's  Inspect hello world example for AI evals
by TheManxLoiner — 2024-05-16T20:47:44.346Z
Link: https://lovkush.medium.com/evaluating-llms-using-uk-ai-safety-institutes-inspect-framework-96435c9352f3

Sharing my detailed walk-through on using the UK AI Safety Institute's new open source package Inspect for AI evals.

Main points:

Package released in early May 2024 is here: https://github.com/UKGovernmentBEIS/inspect_ai
Seems easy to use and removes boiler-plate code. I am new to evals so I do not know what experienced researchers would look for in such a tool. I am curious to know what others think of it!
There is one unusual behaviour around whether what they call 'scorer' should be independent of what they call 'plan'. I raised an issue about this on GitHub and would be very interested to know what others in AI safety community think of this detail.
Feeling (instrumentally) Rational
by Morphism — 2024-05-16T18:56:12.223Z

Contra this post from the Sequences

In Eliezer's sequence post, he makes the following (excellent) point:

I can’t find any theorem of probability theory which proves that I should appear ice-cold and expressionless.

This debunks the then-widely-held view that rationality is counter to emotions. He then goes on to claim that emotions have the same epistemic status as the beliefs they are based on.

For my part, I label an emotion as “not rational” if it rests on mistaken beliefs, or rather, on mistake-producing epistemic conduct. “If the iron approaches your face, and you believe it is hot, and it is cool, the Way opposes your fear. If the iron approaches your face, and you believe it is cool, and it is hot, the Way opposes your calm.”

I think Eliezer is making a type error here. When he says "rational", he is of course talking about epistemic rationality. However, epistemic rationality is a property of beliefs, not emotions. In other words, I can't find any theorem of probability theory which proves that I should feel sad when my expected utility decreases, and happy when it increases.[1]

The only type of "Rationality" emotions can apply to is instrumental rationality, i.e. "the science of winning at life", and the most instrumentally rational emotions don't always stem from beliefs in the intuitive way that Eliezer describes in his sequence post.

Example: Being sad about a high P(doom) can make you less productive at reducing P(doom), as well as incentivize you to self-decieve into a low P(doom).

If I see more good examples in the comments, I will add them.

^

...and fear when there is high probability mass around a future event that will result in a large loss of utility, and anger when another agent causes your utility to go down, and curiosity when you have an opportunity to gain useful information, etc.
Advice for Activists from the History of Environmentalism
by Jeffrey Heninger — 2024-05-16T18:40:02.064Z
Link: https://blog.aiimpacts.org/p/advice-for-activists-from-the-history

This is the fourth in a sequence of posts taken from my recent report: Why Did Environmentalism Become Partisan?

This post has more of my personal opinions than previous posts or the report itself.

Other movements should try to avoid becoming as partisan as the environmental movement. Partisanship did not make environmentalism more popular, it made legislation more difficult to pass, and it resulted in fluctuating executive action. Looking at the history of environmentalism can give insight into what to avoid in order to stay bipartisan.

Partisanship was not inevitable. It occurred as the result of choices and alliances made by individual decision makers. If they had made different choices, environmentalism could have ended up being a bipartisan issue, like it was in the 1980s and is in some countries in Europe and democratic East Asia.

Environmentalists were not the only people making significant decisions here. Fossil fuel companies and conservative think tanks also had agency in the debate – and their choices were more blameworthy than the choices of environmentalists. Politicians choose who they do and do not want to ally with. My focus is on the environmental movement itself, because that is similar to what other activist groups are able to control.

I am more familiar with the history of the environmental movement than with most other social movements. The environmental movement is particularly interesting because it involves an important global issue that used to be broadly popular, but has since become very partisan and less effective at enacting policy in the United States. It nevertheless can be risky to over-update on a single case study. Much of the advice given here has support in the broader social movements literature, but the particulars are based on the history of one movement.

With those caveats aside, let’s look at what we can learn.

Here is a list of advice I have gleaned from this history:

Make political alliances with individuals and institutions in both political parties.
This is the most important advice.
Allying with the Democratic Party might have seemed like a natural choice at the time. Climate scientists might have already leaned left, and so found allying with Democrats to be more natural – although the evidence for this is weak. Al Gore was committed to their cause, and was rapidly building political influence: from Representative to Sena

...