trentmkelly/LessWrong-43k huggingface dataset:
AI Safety Newsletter #3: AI policy proposals and a new challenger approaches
by ozhang — 2023-04-25T16:15:17.227Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-3

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

---

Policy Proposals for AI Safety

Critical industries rely on the government to protect consumer safety. The FAA approves new airplane designs, the FDA tests new drugs, and the SEC and CFPB regulate risky financial instruments. Currently, there is no analogous set of regulations for AI safety.

This could soon change.

President Biden

and

other

members

of

Congress

have recently been vocal about the risks of artificial intelligence and the need for policy solutions.

From guiding principles to enforceable laws. Previous work on AI policy such as the White House

Blueprint for an AI Bill of Rights

and the NIST

AI Risk Management Framework

has articulated guiding principles like interpretability, robustness, and privacy. But these recommendations are not enforceable – AI developers can simply choose to ignore them.

A solution with more teeth could be on its way. Axios

reports

that Senator Chuck Schumer has been circulating a draft framework for AI governance among experts over the last several weeks. To help inform policy making efforts, the Department of Commerce has issued a

request for comments

on how to effectively regulate AI.

The European Union debates narrow vs. general AI regulation. In Europe, policy conversations are centering around the

EU AI Act

. The Act focuses on eight “high-risk” applications of AI, including hiring, biometrics, and criminal justice. But the rise of general purpose AI systems like ChatGPT calls into question the wisdom of regulating only a handful of specific applications.

An

open letter signed by over 50 AI experts

, including CAIS’s director, argues that the Act should also govern general purpose AI systems, holding AI developers liable for harm caused by their systems. Several members from all political blocs of the EU parliament have

publicly agreed

that rules are necessary for “powerful General Purpose AI systems that can be easily adapted to a multitude of purposes.”

Specific policy proposals for AI safety. With politicians promising that AI regulation is coming, the key question is which proposals they will choose to carry forward into law. Here is a brief compilation of several recent

sets

of

policy

proposals

:

Create an AI regulatory body. A

...
EA might systematically generate a scarcity mindset that produces low-integrity actors
by Severin T. Seehrich — 2023-04-25T15:50:00.058Z

Epistemic status: Highly speculative quick Facebook post. Thanks to Anna Riedl for nudging me to share it here anyway.

Something I've noticed recently is that some people who are in a bad place in their lives tend to have a certain sticky sleazy black holey feel to them. Something around untrustworthiness, low integrity, optimizing for themselves regardless of the cost for the people around them. I've met people like that, and I think when others felt around me like my energy was subtly and indescribably off, it was due to me being sticky in that way, too.

Game-theoretically, it makes total sense for people to be a bit untrustworthy while they are in a bad place in their life. If you're in a place of scarcity, it is entirely reasonable to be strategic about where you put your limited resources. Then, it's just reasonable to only be loyal to others as long as you can get something out of it yourself and to defect as soon as they don't offer obvious short-term gains. And similarly, it make sense for the people around you to be a bit weary of you when you are in that place.

And now, a bit of a hot take: I think most if not all of Effective Altruism's recent scandals have been due to low-integrity sticky behavior. And, I think some properties of EA systematically make people sticky.

We might want to invest some thought and effort into fixing them. So, here's some of EA's sticky-people-producing properties I can spontaneously think of, plus first thoughts on how to fix them that aren't supposed to be final solutions:

1. Utilitarianism

Yudkowsky wrote a thing that I think is true:

"Go three-quarters of the way from deontology to utilitarianism and then stop. You are now in the right place. Stay there at least until you have become a god."

Meanwhile, SBF and probably a bunch of other people in EA (including me at times) have gone all four quarters of the way. If there's no upper bound to when it is enough to make the numbers go up, you'll be in a place of scarcity no matter what, and will be incentivized to defect indefinitely.

I think an explicit belief of "defecting is not a good utilitarian strategy" doesn't help here: Becoming sticky is not a decision, but a subtle shift in your cognition that happens when your animal instincts pick up that your prefrontal cortex thinks you are in a place of scarcity.

Basically, I think Buddhism is what utilitarianism would b...
Max Tegmark's new Time article on how we're in a Don't Look Up scenario [Linkpost]
by Jonas Hallgren — 2023-04-25T15:41:16.050Z
Link: https://time.com/6273743/thinking-that-could-doom-us-with-ai/

https://time.com/6273743/thinking-that-could-doom-us-with-ai/

Max Tegmark has posted a Time article on AI Safety and how we're in a "

Don't Look Up

" scenario.

In a similar manner to Yudkowsky, Max went on Lex Fridman and has now posted a Time article on AI Safety. (I propose we get some more people into this pipeline)

Max, however, portrays a more palatable view regarding societal standards. With his reference to Don't Look Up, I think this makes it one of my favourite pieces to send to people new to AI Risk, as I think it describes everything that your average joe needs to know quite well. (An asteroid with a 10% risk of killing humanity is bad)

In terms of general memetics, it will be a lot harder for someone like LeCun to come up with a genius equivalence between

asteroid safety and airplane safety

with this framing. (Which might be a shame since it's one of the dumber counter-arguments I've heard.)

But who knows? He might just claim that scientists know and have always known

how to use nuclear bombs to shoot the asteroid away

or something.

What I wanted to say with the above is that I think Max is doing a great way of framing the problem, and with his respect from his earlier career as a physicist, I think it would be good to use his articles more in public discussions. I also did quite enjoy how he described alignment on the

Lex Fridman podcast

, and even though I don’t agree with all he says, it’s good enough.
A Rant on Calculus III
by Wofsen — 2023-04-25T14:51:40.201Z

I was recently reviewing Calculus III. I had forgotten it since high school, and, to be honest, never really understood it at the time; though I blame the curriculum. This is a guide to learning the key ideas in a couple of hours. Please excuse my strong language, I wrote the following rant in an impassioned moment.

The way that Calculus III is taught today is stupid. The notation for the class is all from electrostatics, and the theory of the class is a single theorem from Riemannian Geometry (Summarized, Explained, Additional Lecture), but this is buried in jargon. Rather than mine for the truth, some bozo slapped a stupid river analogy on top of the junk heap, and all the teachers nodded along and decided that this should become the accepted wisdom. This is a good way to make everyone so unbelievably bewildered that they can’t solve a single problem. Even 3B1B can barely make this analogy coherent. If you must have 'flow' imagine a Snap! freeze-frame of the river where the currents are somehow magically preserved and something still moves around in it. But that’s stupid, things don't move through ice! Any toddler can tell you that frozen water is ice and that ice does not have a 'flow' (a bright youngster might mention glaciers, but at that scale everything's different). You know what these mathematics do describe? How about the thing they were literally designed to describe: magnetism! You know what happens if you freeze molten iron in a magnetic current generated by an electric coil? It hardens into a magnet. That’s how magnets are made! Guess what: magnets pull iron filings despite being STATIC. They are the magic ‘frozen river’ these bozos keep trying to describe. Don’t try and invent some stupid analogy for something that is best described by nature. Look at nature; look to experiment, and you can actually begin to have an idea about how all this shit works. If you’re taking the class today and confused out of your mind as much as I was, I give you this advice: watch some demonstrations of magnetism on YouTube or play (carefully) with some toy magnets. You’ll get infinitely more Bayesian updating playing with magnet toys first and refining with the mathematics of Maxwell's Equations and Stokes's Theorem afterwards.
Briefly how I've updated since ChatGPT
by rime — 2023-04-25T14:47:15.939Z

I'm laying out my thoughts in order to get people thinking about these points and perhaps correct me. I definitely don't endorse deferring to anything I say, and I would write this differently if I thought people were likely to do so.

OpenAI's model of "deploy as early as possible in order to extend the timeline between when the world takes it seriously to when humans are no longer in control" seems less crazy to me.

I think ChatGPT has made it a lot easier for me personally to think concretely about the issue and identify exactly what the key bottlenecks are.
To the counterargument "but they've spurred other companies to catch up," I would say that this was going to happen whenever an equivalent AI was released, and I'm unsure whether we're more doomed in the world where this happened now, versus later when there's a greater overhang of background technology and compute.
I'm not advocating specifically for or against any deployment schedule, I just think it's important that this model be viewed as not crazy, so it's adequately considered in relevant discussions.

Why will LLMs develop agency? My default explanation used to involve fancy causal stories about monotonically learning better and better search heuristics, and heuristics for searching over heuristics. While those concerns are still relevant, the much more likely path is simply that people will try their hardest to make the LLM into an agent as soon as possible, because agents with the ability to carry out long-term goals are much more useful.
"The public" seems to be much more receptive than I previously thought, both wrt Eliezer and the idea that AI could be existentially dangerous. This is good! But we're at the beginning where we are seeing the response from the people who are most receptive to the idea, and we've not yet got to the inevitable stage of political polarisation.
Why doom? Companies and the open source community will continue to experiment with recursive LLMs, and end up with better and better simulations of entire research societies (a network epistemologist's dream). This creates a "meta-architectures overhang" which will amplify the capabilities of any new releases of base-level LLMs. As these are open sourced or made available via API, somebody somewhere will plain tell them to recursively self-improve themselves, no complicated story about instrumental convergence needed.

AI will not s

...
Explaining the Transformer Circuits Framework by Example
by Felix Hofstätter — 2023-04-25T13:45:27.000Z

Acknowledgement: I want to thank Joseph Bloom, Michael Ripa, Tilman Räuker and Alexander Spies for their feedback and comments on the draft of this post. Some of the code for creating visualizations is based on works by Matthew Baggins and Neel Nanda. I am currently supported by a grant from the Long-Term-Future fund without which I would not have had the time to work on this post.

This post is meant for people who are just getting started with mechanistic interpretability and have an understanding of the transformer architecture as well as the underlying linear algebra. Maybe you have not yet read the mathematical framework for transformer circuits (MFTC) or have read it but are still confused about some things. I know that I certainly was: concepts like QK- and OV-circuits seemed to make sense algebraically but I was lacking intuition about how they work in practice. In essence, I am aiming to write a post that would have clarified the confusions I had when starting out and that would have given me the intuitions I wish I had.

As part of skilling up on mechanistic interpretability I tackled two of the beginner problems from Neel Nanda's list of 200 concrete MI problems: exploring how transformers take the max of a list and how they model the Fibonacci sequence. This post starts with a summary of the framework, focusing on the QK- and OV-circuit. Sections 2 and 3 then give a more practical explanation of how these circuits work using examples from the MI problems I worked on. Section 2 is about a single-layer attention-only model which predicts the highest integer from its input sequence as the next token. In this setting we can find the "full" circuits which map from tokens to tokens and can thus be nicely visualized. In the third section I consider a model with a block of fully connected layers (MLP) after the attention block. I explain how the ideas from MFTC can be applied in this setting - this is similar to the derivation of effective weights in A Mechanistic Interpretability Analysis of Grokking.

On terminology: In this post I mainly use "circuit" to denote interpretable, linear subsets of operations in a transformer. This is how the term is used in MFTC to describe the QK- and OV-circuits. More generally, "circuit" is also often used for any subset of a transformer that can be identified as being responsible for a particular task and these...
Notes on Potential Future AI Tax Policy
by Zvi — 2023-04-25T13:30:01.015Z

Response To: Via Marginal Revolution, Brian Slesinsky proposes a tax on language model API calls.

Brian Slesinsky: My preferred AI tax would be a small tax on language model API calls, somewhat like a Tobin tax on currency transactions. This would discourage running language models in a loop or allowing them to “think” while idle.

For now, we mostly use large language models under human supervision, such as with AI chat. This is relatively safe because the AI is frozen most of the time [1]. It means you get as much time as you like to think about your next move, and the AI doesn’t get the same advantage. If you don’t like what the AI is saying, you can simply close the chat and walk away.

Under such conditions, a sorcerer’s apprentice shouldn’t be able to start anything they can’t stop. But many people are experimenting with running AI in fully automatic mode and that seems much more dangerous. It’s not yet as dangerous as experimenting with computer viruses, but that could change.

Such a tax doesn’t seem necessary today because the best language models are very expensive [2]. But making and implementing tax policy takes time, and we should be concerned about what happens when costs drop.

Another limit that would tend to discourage dangerous experiments would be a minimum reaction time. Today, language models are slow. It reminds me of using a dial-up modem in the old days. But we should be concerned about what happens when AI’s start reacting to events much quicker than people.

Different language models quickly reacting to each other in a marketplace or forum could cause cascading effects, similar to a “flash crash” in a financial market. On social networks, it’s already the case that volume is far higher than we can keep up with. But it could get worse when conversations between AI’s start running at superhuman speeds.

Financial markets don’t have limits on reaction time, but there are trading hours and circuit breakers that give investors time to think about what’s happening in unusual situations. Social networks sometimes have rate limits too, but limiting latency at the language model API seems more comprehensive.

Limits on transaction costs and latency won’t make AI safe, but they should reduce some risks better than attempting to keep AI’s from getting smarter. Machine intelligence isn’t defined well enough to regulate. There are many benchmarks and

...
Sentience in Silicon: The Challenges of AI Consciousness
by Hannes Thurnherr — 2023-04-25T13:15:56.358Z

Why it’s important

The current phase of acceleration in AI has increased the importance of the debate around consciousness to a degree that I never thought would arrive this early. Systems like GPT-4 are passing most of our, thus far thought of, tests for consciousness[1]. Still, only very few even consider the consciousness of an LLM to be a possibility. Even if they may have initially agreed with the metrics used to measure consciousness. Why is that?

Intuitively, it makes sense. When I consider a simple logic gate on a computer chip inside a server, responding to changes in electrical currents, I don't associate it with consciousness any more than I do a pebble on the ground. Similarly, if I were to see a brain outside of a human skull, it might not immediately seem conscious, even though we know it is capable of consciousness. Thus, it's evident that our intuition is not a reliable guide for determining whether something possesses consciousness.

Another argument I often hear is simply an appeal to the underlying mechanism. “LLMs cannot be conscious. They are simply neural networks that predict the next token!”.[2] I think this argument is equally flawed. It assumes that mechanical explainability and consciousness are mutually exclusive. We don’t really understand the brain but I can’t think of any serious person who says that we never will. So I don't think that merely knowing the underlying mechanism at its most basic level, be that the binary configurations of transistors, the word predictions of LLMs or the firing of neurons should disqualify anything from being considered conscious.

There is a third major argument for the biological monopoly on consciousness, which I only consider to be worth discussing because one of its major proponents is sir Roger Penrose, the renowned physicist. At its most basic, it is the belief that our consciousness, our subjective experience is the result of a thus far undiscovered property of matter, that only so happens to exclusively exist in the matter of our brains.[3] To me, it seems like a profoundly unscientific approach to take one's own subjective experience, by itself, as evidence, that our extremely well-supported ideas about something as basic as matter, are somehow incomplete or wrong. Especially since cognition otherwise measurably happens on the far bigger scale of large molecules or cells, not subatom...
Paths to failure
by Karl von Wendt — 2023-04-25T08:03:34.585Z

Epistemic status: This is an intermediate result of the discussions between Daniel O’Connell, Ishan, Karl von Wendt, Péter Drótos, and Sofia Bharadia during the ongoing project “Developing specific failure stories about uncontrollable AI” at this year’s

virtual AI Safety Camp

. As a basis for the stories we want to develop in the project, we try to map the territory for different paths to existential risks from uncontrollable AI. This map is preliminary and very likely incomplete. Comments and suggestions for additional paths are welcome.

Key assumptions

While exploring the different paths to failure we maintain the following assumptions:

A sufficiently advanced AI will likely be uncontrollable: This means that it will be able to circumvent or counter all human interventions to correct its decisions or prevent its actions.
Uncontrollability doesn’t require AGI or ASI: To become uncontrollable, the AI doesn’t necessarily have to be an AGI in the sense that it can solve most tasks at a human level, nor does it have to be “superintelligent”. The AI just needs the specific abilities necessary to overcome human intervention, e.g. by hacking technical systems and/or by skillfully manipulating humans and preventing coordination.
Uncontrollability arises before alignment: By the time we can develop an uncontrollable AI, we will likely not know how to give it a provably beneficial goal, so it will by default be misaligned with human values.
Uncontrollability leads to existential catastrophe: Since the number of possible world states the AI could strive for is vastly greater than the possible world states in which human values are maintained, there is a significant probability that a misaligned uncontrollable AI would lead to an existential catastrophe.

Based on these assumptions, we explore some potential paths to failure. We follow these paths only up until the point that an AI becomes uncontrollable, but do not go into the details of what happens post-uncontrollability. What the model chooses to do after humans are unable to stop it depends on the specifics of the original goal given to the AI. Exploring only up until uncontrollability also means that we do not need to differentiate between existential risks and suffering risks, assuming that an uncontrollable, misaligned AI will create a world state sufficiently bad from a human perspective that it is to be avo...
My Assessment of the Chinese AI Safety Community
by Lao Mein — 2023-04-25T04:21:19.274Z

I've heard people be somewhat optimistic about this AI guideline from China. They think that this means Beijing is willing to participate in an AI disarmament treaty due to concerns over AI risk. Eliezer noted that China is where the US was a decade ago in regards to AI safety awareness, and expresses genuine hope that his ideas of an AI pause can take place with Chinese buy-in.

I also note that no one expressing these views understands China well. This is a PR statement. It is a list of feel-good statements that Beijing publishes after any international event. No one in China is talking about it. They're talking about how much the Baidu LLM sucks in comparison to ChatGPT. I think most arguments about how this statement is meaningful are based fundamentally on ignorance - "I don't know how Beijing operates or thinks, so maybe they agree with my stance on AI risk!"

Remember that these are regulatory guidelines. Even if they all become law and are strictly enforced, they are simply regulations on AI data usage and training. Not a signal that a willingness for an AI-reduction treaty is there. It is far more likely that Beijing sees near-term AI as a potential threat to stability that needs to be addressed with regulation. A domestic regulation framework for nuclear power is not a strong signal for a willingness to engage in nuclear arms reduction.

Maybe it is true that AI risk in China is where it was in the US in 2004. But the US 2004 state was also similar to the US 1954 state, so the comparison might not mean that much. And we are not Americans. Weird ideas are penalized a lot more harshly here. Do you really think that a scientist is going to walk up to his friend from the Politburo and say "Hey, I know AI is a central priority of ours, but there are a few fringe scientists in the US asking for treaties limiting AI, right as they are doing their hardest to cripple our own AI development. Yes, I believe they are acting in good faith, they're even promising to not widen the current AI gap they have with us!" Well, China isn't in this race for parity or to be second best. China wants to win. But that's for another post.

Remember that Chinese scientists are used to interfacing with our Western counterparts and know to say the right words like "diversity", "inclusion", and "no conflict of interest" that it takes to get our papers published. Just because ...