trentmkelly/LessWrong-43k huggingface dataset:
Announcing the CNN Interpretability Competition
by scasper — 2023-09-26T16:21:50.276Z

TL;DR

I am excited to announce the

CNN Interpretability Competition,

which is part of the competition track of

SATML 2024

.

Dates: Sept 22, 2023 - Mar 22, 2024

Competition website:

https://benchmarking-interpretability.csail.mit.edu/challenges-and-prizes/

Total prize pool: $8,000

NeurIPS 2023 Paper: Red Teaming Deep Neural Networks with Feature Synthesis Tools

Github:

https://github.com/thestephencasper/benchmarking_interpretability

For additional reference: Practical Diagnostic Tools for Deep Neural Networks

Correspondence to:

interp-benchmarks@mit.edu

Intro and Motivation

Interpretability research is popular, and interpretability tools play a role in almost every agenda for making AI safe. However, there are some gaps between the research and engineering applications. If one of our main goals for interpretability research is to help us align highly intelligent AI systems in high-stakes settings, we need more tools that help us better solve practical problems.

One of the unique advantages of interpretability tools is that, unlike test sets, they can sometimes allow humans to characterize how networks may behave on novel examples. For example,

Carter et al. (2019)

,

Mu and Andreas (2020)

,

Hernandez et al. (2021)

,

Casper et al. (2022a)

, and

Casper et al. (2023)

have all used different interpretability tools to identify novel combinations of features that serve as adversarial attacks against deep neural networks.

Interpretability tools are promising for exercising better oversight, but

human understanding is hard to measure, and it has been difficult to make clear progress toward more practically useful tools

. Here, we work to address this by introducing the

CNN Interpretability Competition

(accepted to

SATML 2024

).

The key to the competition is to develop interpretations of the model that help human crowdworkers discover trojans: specific vulnerabilities implanted into a network in which a certain trigger feature causes the network to produce unexpected output. In addition, we also offer an open-ended challenge for participants to discover the triggers for secret trojans by any means necessary.

The motivation for this trojan-discovery competition is that trojans are bugs caused by novel trigger features -- they usually can’t be identified by analyzin...
Making AIs less likely to be spiteful
by Nicolas Macé — 2023-09-26T14:12:06.202Z

Which forms of misalignment might result in particularly bad outcomes? And to what extent can we prevent them even if we fail at ​​intent alignment? We define spite as a terminal preference for frustrating others’ preferences, at least under some conditions. Reducing the chances that an AI system is spiteful is a candidate class of interventions for reducing

risks of AGI conflict

, as well as

risks from malevolence

. This post summarizes some of our thinking on the topic. We give an overview of why spite might lead to catastrophic conflict; how we might intervene to reduce it; ways in which the intervention could fail to be impactful, or have negative impact; and things we could learn that would update us on the value of this intervention.

Key takeaways

Spiteful preferences include a generalized preference for harming others, as well as other preferences like vengefulness and spite towards certain groups. The basic reason to focus on reducing spite is that such interventions may stably make AIs less likely to take risks of mutually costly conflict (or deliberately create suffering because they intrinsically value it), even if alignment fails. (

more

)
Spite might be selected for in ML systems because (a) it serves as a strategically valuable commitment device, (b) it is a direct proxy for high-scoring behavior in environments where the optimal behavior involves harming other agents (e.g., environments with competition between agents), (c) it is (correctly or incorrectly) inferred from human preferences, or (d) it results from miscellaneous generalization failures. (

more

)
Thus potentially low-cost interventions to reduce the chances of spite include modifications to the training data or loss function to reduce selection pressure towards spite (e.g., avoiding selecting agents based on relative performance in multi-agent tasks, or filtering human feedback that could select for spite). (

more

)

Reducing spite carries some prima facie backfire risk, via potentially increasing the exploitability of AIs that share human values to some extent. We currently don’t think that this consideration makes the sign of spite reduction negative, however. One reason is that, for interventions to backfire by making our AI more exploitable, they have to change other agents’ beliefs about how spiteful our AI is, and there are various reasons to doubt this will

...
[Linkpost] Mark Zuckerberg confronted about Meta's Llama 2 AI's ability to give users detailed guidance on making anthrax - Business Insider
by mic — 2023-09-26T12:05:57.396Z
Link: https://www.businessinsider.com/mark-zuckerberg-senate-ai-forum-llama-2-gives-anthrax-instructions-2023-9

Several tech leaders descended upon Capitol Hill last week to discuss the rapid expansion of generative AI. It was a mostly staid meeting until the potential harms from Meta's new Llama 2 model came up.

During the discussion, attended by most of the Senate's 100 members, Tristan Harris, a co-founder of the Center for Humane Technology, said he recently had engineers take Meta's powerful large language model

Llama 2

for a "test drive." After some prompting, Harris said that a chat with Llama 2 came back with a detailed walkthrough of how to create anthrax as a biological weapon, according to one person familiar with the forum and two senators present. That prompted a testy exchange between Harris and Mark Zuckerberg, co-founder and CEO of Meta, formerly known as Facebook. Most specifics of the exchange between Harris and Zuckerberg have not been previously reported, although Harris receiving directions from Llama 2 about an unidentified biological weapon was noted by The Washington Post.

Among the

two dozen tech leaders

at the forum were Elon Musk, owner of Twitter and CEO of Tesla and SpaceX; Sam Altman, CEO of OpenAI; Satya Nadella, CEO of Microsoft; Jensen Huang, CEO of Nvidia; and Sundar Pichai, CEO of Google.

The gathering was led by Senate Majority Leader Chuck Schumer, Democratic Sen. Martin Heinrich, and Republican Sens. Mike Rounds, and Todd Young, who all make up a new "artificial intelligence working group." The group formed earlier this year, a few months after OpenAI's ChatGPT bot became known the world over.

During the session, Zuckerberg attempted to downplay Harris' statement that Llama 2 can tell users how to make anthrax, saying anyone who was looking for such a guide could find out how to make anthrax on YouTube, according to both of the senators present. Harris rejected the argument, saying such guides do not come up on YouTube, and even if they did, the level of detail and guidance provided by Llama 2 was unique to such a powerful generative AI model. It's also largely an open-source model, meaning it's freely available to use and adapt.

"It was one of the only moments in the whole thing that was like, 'Oh,'" one of the senators present said, describing the exchange as having caught people's attention. "Twenty-four out of the 26 panelists there basically said exactly the same thing over and over: 'We need to protect AI innovation but with safe

...
Enforcing Far-Future Contracts for Governments
by FCCC — 2023-09-26T04:26:46.442Z

(This post has been strongly downvoted by Jiro. All other votes appear to have been positive.)

I recently made a post titled Far-Future Commitments as a Policy Consensus Strategy. Read that first if you'd like. But the basic idea is that we can pass otherwise unpassable policies by agreeing today that we will enforce them in the future. (In that article, I said a 100-year delay, but I probably should have made it 50 years.)

Are these contracts enforceable? Yes.

Most commenters believed this delay to be unworkable. Surely there’s no way to force a government to hold to promises made 50 years ago. Right?

Well, here’s my attempted solution to the reneging problem in the case of land value taxes.

Let’s say the government issues $1 trillion of what are essentially inflation-adjusted perpetuities, except upon the institution of the delayed law, they cease to pay out. (Unless the law is instituted before the 50-year period, in which case, they only cease to pay out after 50 years.) If the law is not instituted, then they continue to be perpetuities.

So the future government can either enact the policy and get their debt wiped for free, or they can continue to pay the debt. If reneging is guaranteed, we’re essentially just selling regular perpetuities. No big deal; we pay a fair interest rate. If there’s a no chance of reneging, we’re selling a kind of annuity, and we will still pay a fair interest rate.

And there is basically no chance of reneging: name a government that wouldn’t accept the offer to wipe their debt without a default. So at no cost to ourselves, we can offer the future government a trillion dollars, but only if they do the right thing. All we do is issue “enforcement perpetuities” instead of issue regular debt, whenever today’s government needs liquidity.

But let’s say the worst-case scenario occurs: we actually sold perpetuities (the future government is going to renege) but the market thought we were selling a 50-year annuity, so they paid less than it was worth. How much less? Since cashflows occurring more than 50 years into the future are discounted by at least 1+d to the 50th power, not very much. As unlikely as it is to ever happen, it’s not that bad of a financial mistake on our part.

So if everything goes to plan, aren’t the landowners losing money here? Again, not really, due to the money being discounted by 50 years. But if you want to correct t...
A few Alignment questions: utility optimizers, SLT, sharp left turn and identifiability
by Igor Timofeev — 2023-09-26T00:27:23.229Z

I am trying to find some Alignment direction that could be interesting for me to learn and work on, I've already received a couple of answers but I want more opinions and I just get more motivation from the discussions/cooperation.

1. As I understand that almost any agent can be modeled as an utility optimizer, but I am still suspicious about the concept. I am not sure that utility optimization is the most compressed representation of any agent, because an arbitrary utility function could be arbitrarily complex to encode, and maybe there is another model of this agent that is more compressed. Humans seem to be not explicit utility optimizers: Shard Theory. Can we formally distinguish between an explicit utility optimizer and an agent behavior that could be understood as optimization of some function? What is more efficient? Can you please recommend something that answers the question above or is just a good starting point to learn about utility optimizers via a math approach? For example, I want to understand more about existing theorems on the topic, like the Von Neumann–Morgenstern utility theorem.

2. I am trying to find some approaches to AI Alignment that use rigorous math about deep learning. But I am struggling to find even good books about the math behind deep learning as a whole, not about how to do deep learning in practice, but with theorems and proofs. One example I found interesting is "Deep Learning Architectures A Mathematical Approach", chapters about "Approximation Theorems", "Information Processing" and "Geometric Theory". In the Alignment field, I only see SLT theory which tries this path. Maybe someone knows other approaches/books in that direction? If SLT is the only one, is it because it covers all learning theory?

3. If I understand correctly, many results show that neural networks lack identifiability of weights or even latent representations like in VAE. What, then, is the point of Mech Interp, which, as I understand, is trying to understand the function of specific neurons/features? What should I read to understand this? Does it contradict with SLT take that all sophisticated models are singular?

4. I had read papers "Advances in Identifiability of Nonlinear Probabilistic Models" by Ilyes Khemakhem, "On Linear Identifiability of Learned Representations" by Kingma and similar, and immediatly thought why...
Impact stories for model internals: an exercise for interpretability researchers
by jenny — 2023-09-25T23:15:29.189Z

Inspired by Neel's longlist; thanks to @Nicholas Goldowsky-Dill and @Sam Marks for feedback and discussion, and thanks to AWAIR attendees for participating in the associated activity.

As part of the Alignment Workshop for AI Researchers in July/August '23, I ran a session on theories of impact for model internals. Many of the attendees were excited about this area of work, and we wanted an exercise to help them think through what exactly they were aiming for and why. This write-up came out of planning for the session, though I didn't use all this content verbatim. My main goal was to find concrete starting points for discussion, which

have the right shape to be a theory of impact
are divided up in a way that feels natural
cover the diverse reasons why people may be excited about model internals work

(according to me[1]).

This isn't an endorsement of any of these, or of model internals research in general. The ideas on this list are due to many people, and I cite things sporadically when I think it adds useful context: feel free to suggest additional citations if you think it would help clarify what I'm referring to.

Summary of the activity

During the session, participants identified which impact stories seemed most exciting to them. We discussed why they felt excited, what success might look like concretely, how it might fail, what other ideas are related, etc. for a couple of those items. I think categorizing existing work based on its theory of impact could also be a good exercise in the future.

I personally found the discussion useful for helping me understand what motivated some of the researchers I talked to. I was surprised by the diversity.

Key stats of an impact story

Applications of model internals vary a lot along multiple axes:

Level of human understanding needed for the application[2]

If a lot of human understanding is needed, does that update you on the difficulty of executing in this direction? If understanding is not needed, does that open up possibilities for non-understanding-based methods you hadn't considered?
For example, determining whether the model does planning would probably require understanding. On the other hand, finding adversarial examples or eliciting latent knowledge might not involve any.

Level of rigor or completeness (in terms of % model explained) needed for the application

If a high level of rigor or completeness is needed, does that upd

...
Autonomic Sanity
by Sable — 2023-09-25T22:37:07.262Z
Link: https://affablyevil.substack.com/p/autonomic-sanity

Epistemic Status: self-reported musings

Defining Sanity

Mental health is a complicated topic, and “sanity” can be a loaded word, so I’ll offer a few definitions to make sure we’re all on the same page.

The colloquial definition of sanity, if such a thing exists, is formed in contrast to insanity. Someone is sane who is not insane.

So what’s insanity?

Insanity

The colloquial definition of insanity is repeating the same act over and over again and expecting different results.

The dramatic definition of insanity generally involves seeing and/or hearing things that aren’t there, laughing hysterically at nothing, and getting punched in the face by Batman.

My personal definition of insanity is not getting out of bed for two days, not answering your family’s and/or friends’ calls because you’re irrationally terrified of talking to other people, and letting everyone in your life down, including yourself, as you desperately try to pretend that everything’s fine and you’re not a huge screw-up who drags anyone who gets close to you into your nonsense.

So sanity for me is defined as…not that. But I’ll elaborate, because pointing at one particular kind of crazy and shaking my head is not particularly illustrative.

A Personal Definition of Sanity

Sanity, to me, is about being able to live your life roughly the way you want to, without too many internal mental problems. It isn’t about hating your job or being unhappy in your relationship; it’s about being a functional human being most of the time.

At first glance, this does not seem like a high bar to clear, yet it is one I have repeatedly failed to meet at many points in my life.

So what gives? Was I born defective, victim of a genetic imbalance in brain chemicals?

I look at the people in the world around me, and they don’t seem to be constantly clawing their way out of a pit of non-functionality and failure to successfully human. (Not that I always looked that way when I was; “keeping up the pretense” is a well-honed skill I suspect many of us have.)

Still, the question has followed me for years, and recently I gained a new perspective on the issue I’m keen to share.

Autonomic Health

Most of our bodily functions are autonomic[1] - while they can be and often are influenced by our behaviors, our bodies mostly take care of themselves.

Think about it: when was the last time you had to actively support your heart in pumping blood through...
What is wrong with this "utility switch button problem" approach?
by Donald Hobson — 2023-09-25T21:36:47.166Z

Suppose you have some positive utility functions A,B , mathematically considered to be random variables dependant on choice of policy.

Let K be a random variable that is 1 if the button is pressed, 0 otherwise. Again dependant on choice of policy. I am considering that there is a particular time that the button must be pressed to make this a single bit. You have a single chance to switch this AI between 2 utilities, and whatever you pick in that instant is baked in forevermore.

Let a=E(A(1−K)), b=E(BK) be the expected partial utilities.

Now consider all policies. In particular, consider the pareto frontier between a and b. Through the introduction of randomness, we can make that parito frontier continuous.

We want some policy off that frontier.

There will be some policy ha that throws all resources at maximizing a. And there will be some (usually different) policy hb that throws all available resources at maximizing b.

let q be a tradeoff rate. And u=a(1−q)+bq. The parito frontier can now be defined as the policies that maximize u for varying q.

For ha, this is the policy that is optimal when q=0 which has u(ha)=a(ha). Then dudE(K)<0.

Likewise at hb with q=1 have dudE(K)>0.

So somewhere in the middle, by the continuity granted by stochastic mixes of policies, their must be at least one point where dudE(K)=0. Use that policy, or stochastic mixture of policies.

This agent will never pay ϵ utility to set the value of the button one way or the other. Because the policy is one in which has E(u|K=0)=E(u|K=1) (I think) Which hopefully means that this policy does worse than the policy that doesn't pay to change the button, but otherwise does everything else the same.

Put another way, if you pay to flip the button. Then you must care about the state of the button, if the chance the button is pressed changes, you are getting more or less for your ϵ utility. So the gradient can't be 0.

And this is described as a choice of policy. A framework which automatically includes the actions of all sub agents.

So, how does this fail? Who can break this agent and get it to do crazy things?
You should just smile at strangers a  lot
by chaosmage — 2023-09-25T20:12:56.907Z

This is so basic I feel stupid for saying it in such an intellectual place. But most of you guys share my enthusiasm for effective altruism, and here's something really altruistic and really effective that I've been doing for two decades and I think lots of you should do too.

Just smile at strangers. Walk down the street with a default smile on your face, seek eye contact and just keep smiling. You don't need to do anything else, not talk to them or anything.

This is tiny, but it's a pretty unadulterated good thing. Facial feedback research is clear, smiling does make you feel better. It costs basically nothing, and definitely less than you get back in good feelings for yourself. It's a quality of life improvement for everyone involved.

After what has to be hundreds of thousands of such micro-interactions, I can report that nearly half of the strangers will smile back, although often they'll look away as they do. This is probably highly dependent on the culture where you do this, especially on whether it is a high-trust environment.

I have learned not to smile at beggars, though. They usually took it as an invitation to target and haggle me.

To be fair, I'm a male and I don't know if a female doing this would get very different reactions. Maybe some of you want to try it and report back?
The King and the Golem
by Richard_Ngo — 2023-09-25T19:51:22.980Z
Link: https://narrativeark.substack.com/p/the-king-and-the-golem

Long ago there was a mighty king who had everything in the world that he wanted, except trust. Who could he trust, when anyone around him might scheme for his throne? So he resolved to study the nature of trust, that he might figure out how to gain it. He asked his subjects to bring him the most trustworthy thing in the kingdom, promising great riches if they succeeded.

Soon, the first of them arrived at his palace to try. A teacher brought her book of lessons. “We cannot know the future,” she said, “But we know mathematics and chemistry and history; those we can trust.” A farmer brought his plow. “I know it like the back of my hand; how it rolls, and how it turns, and every detail of it, enough that I can trust it fully.”

The king asked his wisest scholars if the teacher spoke true. But as they read her book, each pointed out new errors—it was only written by humans, after all. Then the king told the farmer to plow the fields near the palace. But he was not used to plowing fields as rich as these, and his trusty plow would often sink too far into the soil. So the king was not satisfied, and sent his message even further afield.

A merchant brought a sick old beggar. “I met him on the road here, and offered him food, water, and shelter. He has no family, and only a short time left to live, during which I will provide for his every need. He has nothing to gain from betraying me; this is what allows true trust.” A mother brought her young daughter. “I’ve raised her to lack any evil in her heart, to say only good words and do only good deeds. As long as she is not corrupted, she will remain the most trustworthy in the kingdom.”

The king asked the beggar, “How did you end up in such dire straits?” The beggar let out a sigh, and recounted his sorrows: the neighbors who refused to help him when his crops failed; the murder of his son by bandits as they traveled to a new town; the sickness that took his wife as she labored for a pittance in squalid conditions.

“So you have been wronged?” the king asked.

“Very surely”, the beggar said.

“I will give you revenge on the ones who have wronged you, then. All I ask is for you to denounce this merchant.” The beggar’s decision did not take long—for the trust that came easily was broken easily too.

To the mother, the king asked: “How did you raise such a child? Has she never once strayed?”

“Well, once or twice. But I discipline her fi...