trentmkelly/LessWrong-43k huggingface dataset:
Stopping dangerous AI: Ideal lab behavior
by Zach Stein-Perlman — 2023-05-09T21:00:19.505Z

Disclaimer: this post doesn't have the answers. Moreover, it's an unfinished draft. Hopefully a future version will be valuable, but that will only occur if I revise/rewrite it. For now you're better off reading sources linked from

Ideas for AI labs: Reading list

and

Slowing AI: Reading list

.

Why "stopping dangerous AI" rather than "slowing AI" here? It's more appropriate for actors' actions. Just "slowing" is a precise/continuous variable and what strategists think about while "stopping" is a natural/simple thought and what actors naturally aim for. Note that in the context of slowing, "AI" always meant "dangerous AI."

Blackbox AI safety (and how to do good with powerful AI) to focus on the speed of AI progress. What should leading labs do to facilitate a safe transition to a world with powerful AI? What would I do and plan if I controlled a leading lab?[1]

It would be better if progress toward dangerous AI capabilities was slower, all else equal. (This will become much more true near the end.) Or: it would be better if dangerous AI capabilities appear later, all else equal. (And slowing near the end is particularly important.)

Ways labs can slow AI:

Pause progress toward dangerous systems

And convince others to pause
And push for and facilitate a mandatory pause, e.g. enforced by governments (non-government standards-setters and industry self-regulation are also relevant)
And pause later

Publish less research relevant to dangerous systems

And convince others to publish less
And push for and facilitate mandatory publication rules, e.g. enforced by governments
And decrease diffusion of ideas more generally

Infosec, opsec, cybersec
Deploy slowly and limit API access as appropriate

Raise awareness of AI risk; look for and publicize warning signs; maybe make demos of scary AI capabilities. Influence other labs & government &c– make them better informed about AI risk and how they can help slow AI. Help government stop dangerous AI.
Prepare to coordinate near the end

Commit to slow down near the end
Commit not to compete near the end
Gain the ability to make themselves selectively transparent

Maybe benefit-sharing mechanisms like better versions of the "windfall clause" and creating common knowledge of shared values to reduce incentives to race

(There are lots of other things labs should do for safety– this is just about slowing AI.)

(Note that what it wou...
Progress links and tweets, 2023-05-09
by jasoncrawford — 2023-05-09T20:22:29.339Z
Link: https://rootsofprogress.org/links-and-tweets-2023-05-09

Announcements

Stripe Press is publishing Poor Charlies Almanack (via @patrickc). Also, Londoners: sign up for events from Stripe Press and Works in Progress (via @WorksInProgMag)
Dwarkesh Patel’s podcast The Lunar Society now taking applications for advertisers. Also, Dwarkesh would really love to interview Robert Caro

Links

The Ascent of Man turns 50. Here’s the premiere (via @BBCArchive & @drbronowski)
Celebrating Smallpox Eradication Day
How photographs could be sent by telephone in the 1930s (via @asallen)
Brian Potter on nuclear power plant construction costs (via @calebwatney)
The Hunterian Museum, on anatomy & the history of surgery (via @salonium)

Quotes

Sears was basically Amazon one century earlier
“Humanities bros need to take more STEM classes”
Child labor was once seen as a good thing
Evidence for the use of water wheels early in the first century
Autopilots have been on airplanes for a long time
The Pasteur Institute was privately funded
Something to keep in mind the next time you contact customer service

Queries

What are the best reasons for optimism about AI risk? (@MaxCRoser)
What lessons can we learn about reasoning from Bayes’ rule?
What are the most impressive things we could build if we all really wanted it?
What to read to understand the history of the US electrical grid? (@_brianpotter)
What is going to enable $1/day continuous hormone monitoring?
Is there progress-studies work on the home/family dimensions of flourishing?
What’s the best way to help someone learn to understand another’s POV?
What are some of the most useful impossibility proofs?
Who is going to create the Cshirchive?

Tweets

John Carmack: “the world is better than it has ever been… and the derivative is still positive… get on board the progress train.” Fortunately the malaise is not universal
Sam Altman’s “alternative path for society”
Dustin Moskovitz: “By 2030, I expect ~everyone will have a personal AI agent”
India building nuclear at < $2/W?
How LLMs can be used for causal inference
Could protein folding be ready for its fundamental theory?
Life in America 40 years ago vs. today
Adam Thierer’s progress bookshelf
Robotaxis were perhaps over hyped in 2017, but opposite is true now
Until 150 years ago people lived in a permanent power outage
We accept “the community decides” for land use, but not elsewhere
A neutron star is like a cargo ship crunched down to the size of a cake sprinkle
After all

...
Have you heard about MIT's "liquid neural networks"? What do you think about them?
by Ppau — 2023-05-09T20:16:27.280Z

I came across this video by MIT CSAIL.

Here is the article they are talking about: https://www.science.org/doi/10.1126/scirobotics.adc8892

This team claims to have achieved driving tasks that previously required 10000 neurons, while using only 19, by using "liquid neural networks" inspired by worm neurology.

They say this innovation brings massive improvements on performance, especially in embedded systems, but also in interpretability, since the reduced number of neurons makes the system much more human-readable. In particular, the attention of the system would be much more easily tracked; this would open the door to safety certifications for high-stakes applications.

Having tried driving and flying tasks in different conditions and environments, they also claim that their system is vastly better at out-of-distribution zero-shot tasks.

So basically, they believe they have made very substantial steps in pretty much every dimension that matters, both for performance and for safety.

As far as I can tell these are very serious researchers, but doesn't that sound a bit too god to be true? I have no expertise in machine learning and I haven't seen any third-party opinions on this yet, so I'm having a hard time making up my mind.

I'd be curious to hear your takes!
Respect for Boundaries as non-arbirtrary coordination norms
by Jonas Hallgren — 2023-05-09T19:42:13.194Z

Introduction

Andrew Critch has recently written about how respect for boundaries are norms that,

to some extent, should convergently arise

.

In this post, I want to link the concept of boundary preservation to active inference and, more importantly, game theory to show that it indeed makes sense for boundaries to be preserved over time.

We will look at Daniel Dennett’s reasoning for humans wanting an intact sense of “free will” as an example of a boundary.

An important point for the entire conversation of boundaries is that it is implicitly contained within an evolutionary or

decision-theoretic

frame. So we will go one meta-level up and talk about the optimality for individuals within environments with or without boundary-specific coordination norms. We will conclude that boundaries are isomorphic to coordination norms in multi-agent iterated prisoner's dilemmas.

Negotiation & Free Will

In his book Intuition Pumps and other tools for Thinking, Daniel Dennett asks why free will is such an essential concept for humans. The answer that he comes up with goes something like the following:

Imagine you’re selling a computer to someone else; you bought it for 1000$ two years ago and want somewhere around 500-800$ for it. The other person is willing to buy it for 400-600$ depending on what you give as an initial offer. In a normal situation, you might get a price of 550$ for the computer.

Let’s now imagine that the person you’re negotiating with has a great sense of social cues and notices that you’re happy about the trade. They might now push you even further and get the price lower. Why did this happen? They got a glimpse into the decision process in your head so they could model you and predict what price you would have wanted.

They were then able to predict what you were going to do in the future.

This is an example relevant to what Dennett believes the entire fuzz in the debate on free will is about, a boundary that retains our internal thoughts or, in <<Boundaries>> speak, defends our

viscera

from infiltration from the external world.

Angle Shooting In Poker

Let’s now look at another real-world example of this happening and what relation that has with boundary preservation.

In poker, there’s a phenomenon known as “

angle shooting

”; this is when someone does something deceptive to gain informat...
Solving the Mechanistic Interpretability challenges: EIS VII Challenge 1
by StefanHex — 2023-05-09T19:41:10.528Z

We solved the first (edit: and second) Mechanistic Interpretability challenge that Stephen Casper posed in EIS VII. We spent the last Alignment Jam hackathon attempting to solve the two challenges presented there, and present our (confirmed) solution to the CNN challenge here. We present a write-up of our work on the Transformer challenge in this follow-up post.
Stefan and Marius submitted an early version of this work at the end of the Hackathon, and Stefan added Intervention and Causal Scrubbing tests to the final write-up. A notebook reproducing all results is provided here (requires no GPU but ~13 GB RAM).

The challenges each provide a pre-trained network, and the task is to reverse engineer the network as well as to infer the labeling function used for training. The first challenge network is a MNIST CNN that takes MNIST images and outputs labels. The hints given are that [1] the labels are binary, [2] the test set accuracy is 95.58%, [3] that the (secret) labeling function is simple, and [4] this image:

Hint 4: clue_image

The MNIST network consists of

2 Convolutional layers (Conv -> ReLU -> Dropout -> Pool)x2
2 Fully connected layers (fc1[400,200] -> ReLU -> fc2[200,2])

and we can access the data (torchvision.datasets.MNIST) but not the ground truth labels.

Spoilers ahead!

Summary of our solution (TL,DR)

The inputs are labelled based on similarity with a 1 versus similarity with an inverted 1 ("anti-1"). If the difference is large (either clearly 1 or clearly anti-1) the image is labeled as class 0, otherwise the image is labeled as class 0. Specifically, the template for 1 seems to be is the given hint (clue_image), and the "anti-1" is 1-clue_image.
The similarity is measured as the sum over the element-wise product of the image matrices (or equivalently dot product of flattened image arrays).
Then the the ~17k most similar images to “1” and ~14k most similar images to “anti 1” are labelled class 1, the remaining ~29k images are labelled class 0. We can also phrase this as a band filter for similarity with (clue_image - 0.5), defining class 0 as where -17305 < (image * (clue_image - 0.5)).sum() < -7762.
We can observe this internally by looking at the embedding in the 200-neurons space, a PCA decomposition colored by label shows how the model judges this similarity:
The model internally implem

...
Forecasting as a tool for teaching the general public to make better judgements? 
by Dominik Hajduk | České priority — 2023-05-09T17:35:42.558Z

Tl;dr: We are wondering whether there might be more value in forecasting as a popularized educational activity than seems to be the consensus. Looking for explicit opinions and feedback.

We are a team working on popularizing evidence- and rationality-based approaches in the Czech Republic, including forecasting. When considering and consulting our previous and upcoming projects, we came to realize that there seems to be too little attention paid to forecasting as a skill-building exercise.

In this sense, we agree with David Althaus and kokotajlod’s older

post

regarding the mainstreaming of forecasting. While it’s not society’s biggest issue, we identify especially with the second of their points, which can be expanded as follows:

Forecasting builds an explicit link from its participants to a ground truth which can (with varying success, depending on the context) be objectively claimed.
This link can serve as a grounding rod for rationalization efforts which can be viewed as (otherwise) value-neutral.

What’s more, forecasting principles can be used and trained across various domains to meet people wherever they are most open to learning - from geopolitical questions to very personal ones. Additionally, using probabilistic forecasting in personal questions, some important aspects of rationality can be internalized quite effectively. I point to Julia Galef’s

approach

of satisficing using subjective criteria and qualitative probability expressions to get beyond the initial issue of not having any predictions at all.

At the same time, this satisficing need not be a significant drawback to the benefits that could flow from a wider diffusion of forecasting. Whether it be

depolarization

in politics or laying the groundwork for an

epistemic infrastructure

so that we can more consciously

build shared mental models

, the investment necessary to start reaping rewards could be quite small. Most concretely, this prior is based on the results from

Chang et al.

- we think positive spillovers could be achieved even with modest interventions. In fact, just sticking with the previously referenced post, already types 13-16 in Julia Galef’s taxonomy could all have positive societal externalities if taken up more broadly.

Of course, the answer to the implicit question which seems to be rearing its head:

“Why aren’t we trying to make pe...
Asimov on building robots without the First Law
by rossry — 2023-05-09T16:44:16.957Z

From Caves of Steel_, pp. 160-161 in my version._

“Why can’t a robot be built without the First Law? What’s so sacred about it?”

Dr. Gerrigel looked startled, then tittered, “Oh, Mr. Baley.”

“Well, what’s the answer?”

“Surely, Mr. Baley, if you even know a little about robotics, you must know the gigantic task involved, both mathematically and electronically, in building a positronic brain.”

“I have an idea,” said Baley. He remembered well his visit to a robot factory once in the way of business. He had seen their library of book-films, long ones, each of which contained the mathematical analysis of a single type of positronic brain. [...] Oh, it was a job, all right. Baley wouldn’t deny that.

Dr. Gerrigel said, “Well, then, you must understand that a design for a new type of positronic brain, even one where only minor innovations are involved, is not the matter of a night’s work. It usually involves the entire research staff of a moderately sized factory and takes anywhere up to a year of time. Even this large expenditure of work would not be nearly enough if it were not that the basic theory of such circuits has already been standardized and may be used as a foundation for further elaboration. The standard basic theory involves the Three Laws of Robotics: the First Law, which you’ve quoted; the Second Law which states, ‘A robot must obey the orders given by human beings except where such orders would conflict with the First Law,’ and the Third Law, which states, ‘A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.’ Do you understand?”

R. Daneel, who, to all appearances, had been following the conversation with close attention, broke in. “If you will excuse me, Elijah, I would like to see if I follow Dr. Gerrigel. What you imply, sir, is that any attempt to build a robot, the working of whose positronic brain is not oriented about the Three Laws, would require first the setting up of a new basic theory and that this, in turn, would take many years.”

The roboticist looked very gratified. “That is exactly what I mean, Mr.…”

Baley waited a moment, then carefully introduced R. Daneel: “This is Daneel Olivaw, Dr. Gerrigel.”

“Good day, Mr. Olivaw.” Dr. Gerrigel extended his hand and shook Daneel’s. He went on, “It is my estimation that it would take fifty years to develop the basic theory of a non-Asenion positr...
Making Up Baby Signs
by jefftk — 2023-05-09T16:40:03.856Z

Babies are often mentally ready to start talking before they have
enough control over their mouths, at a time when they may have enough
control over their arms and hands to make simple gestures; a lot of
otherwise verbal families have had good experiences with signing. If
you look around online for baby sign resources, however,

most of

what

you'll

find involves long
lists of specific signs, often simplified versions signs from American
Sign Language.

If you're just going to have a dozen signs that you use for a few
months, however, a lot of the constraints that real sign languages
like ASL have developed around aren't relevant. You're not building
the foundation for something deep and expressive, you're developing a
pidgin with your toddler to help fill a short gap in their
capabilities. Which means you can prioritize the specific things your
child wants to talk about, and use signs that are very easy for them
to create and distinguish.

For example, with our youngest we ended up with, in rough order of
acquisition and maybe forgetting some:

"more": bring hands together twice

"light": open and close hand twice

"diaper": tap side of butt twice

"drink": tap mouth twice

"pickle": tap side of head twice

"peanut butter": tap inside of elbow twice

"ice cream": tap bottom of chin twice

"bran flakes": tap shoulder twice

"all done": arms in a "w", wave hands twice

"lift me up here": tap surface twice

"outside": raise fist above head

"umbrella": tap palm twice

These are a mixture of ones we taught her ("more", "light", "drink"),
ones she made up ("outside", "lift me up here"), and ones where we
started teaching her something but what she ended up actually doing
was pretty different ("all done", "ice cream"). Since all that
matters is that we can understand each other, noticing what she's
doing and going with that is fine.

It's not a coincidence that many of these are "double tap somewhere".
Once I realized she was picking those up well I started choosing that
form for new signs. It's like, if your child was only going to use
verbal speech for a short transition phase and no one in their life
already used speech, you might use a lexicon like "baba" / "dada" /
"mama" / "nana" / "gaga", allocated to whatever concepts they were
most excited about expressing.

She learned most of these signs she around 17m, during a period wh...
Exciting New Interpretability Paper!
by research_prime_space — 2023-05-09T16:39:35.009Z

There's a pretty exciting new interpretability paper, which hasn't really received the requisite attention because it's not billed as such.

This paper modifies the transformer architecture so that a forward pass minimizes a specifically engineered energy function.

According to the paper, "This functionality makes it possible to visualize essentially any token representation, weight, or gradient of the energy directly in the image plane. This feature is highly desirable from the perspective of interpretability, since it makes it possible to track the updates performed by the network directly in the image plane as the computation unfolds in time".

They achieve SOTA on two of the domains they tested on, although they didn't test on NLP or CV tasks (which is why the paper was rejected, I believe the authors will resubmit again with more experiments.)

More generally, I think architectures such as the above that essentially give you interpretability for free are a promising research direction.
Result Of The Bounty/Contest  To Explain Infra-Bayes In The Language Of Game Theory
by johnswentworth — 2023-05-09T16:35:26.751Z

A month and a half ago I announced a $500 bounty/prize to explain infrabayes work in the language of game theory. Others added another $980 to the prize pool.

There were no entries which did exactly the thing I asked for, but we did get one post which I think provided an excellent 80/20. Or, well, almost 80/20, as implied by the payout: I have decided to pay out 75% of the prize money to David Matolcsi's A very non-technical explanation of the basics of infra-Bayesianism.

David's post is the best conceptual explanation I have seen to date of the core ideas of infra-bayes. It is the only intro I've seen which presents things in a significantly different conceptual frame/language than the original, which I think is very much necessary in order to understand what's going on. After reading it, I feel like I have a decent intuitive idea of what infrabayes does/doesn't do, and roughly how/why it does/doesn't do those things.

The main thing David's post does not do is explicitly tie any of the concepts to the math. Or use any math at all. Which is a reasonable choice - as I said, 80/20(ish).

Thankyou David for writing a very useful post! Again, the post is A very non-technical explanation of the basics of infra-Bayesianism, and I strongly recommend it for anyone who wants an intuitive understanding of what's going on with infrabayes.
AI Safety Newsletter #5: Geoffrey Hinton speaks out on AI risk, the White House meets with AI labs, and Trojan attacks on language models
by Dan H — 2023-05-09T15:26:55.978Z
Link: https://newsletter.safe.ai/p/ai-safety-newsletter-5

Welcome to the AI Safety Newsletter by the

Center for AI Safety

. We discuss developments in AI and AI safety. No technical background required.

Subscribe here to receive future versions.

---

Geoffrey Hinton is concerned about existential risks from AI

Geoffrey Hinton won the Turing Award for his work on AI. Now he says that part of him regrets his life’s work, as he believes that AI poses an existential threat to humanity. As Hinton puts it, “it’s quite conceivable that humanity is just a passing phase in the evolution of intelligence.”

AI is developing more rapidly than Hinton expected. In 2015, Andrew Ng argued that worrying about AI risk is like worrying about

overpopulation on Mars

. Geoffrey Hinton also used to believe that advanced AI was decades away, but recent progress has changed his views. Now

he says that

AI will become “smarter than a human” in “5 to 20 years, but without much confidence. We live in very uncertain times.”

The AI race is heating up, but Hinton sees a way out. In an

interview with MIT Technology Review

, Hinton argues that building AI is “inevitable” given competition between companies and countries. But he argues that “we’re all in the same boat with respect to existential risk,” so potentially “we could get the US and China to agree like we could with nuclear weapons.”

Similar to climate change, AI risk will require coordination to solve. Hinton compared the two risks by

saying

, "I wouldn't like to devalue climate change. I wouldn't like to say, 'You shouldn't worry about climate change.' That's a huge risk too. But I think this might end up being more urgent."

When AIs create their own subgoals, they will seek power. Hinton argues that AI agents like

AutoGPT and BabyAGI

demonstrate that people will build AIs that choose their own goals and pursue them. Hinton and

others

have argued that this is dangerous because “getting more control is a very good subgoal because it helps you achieve other goals.”

Other experts are speaking up on AI risk.

Demis Hassabis

, CEO of DeepMind, recently said that he believes some form of AGI is “a few years, maybe within a decade away” and recommended “developing these types of AGI technologies in a cautious manner.” Shane Legg, co-founder of DeepMind, thinks AGI is likely to arrive around 2026.

Warren Buffet

compared AI to the nuclear bomb, and many others are

concerned about advanced AI

.

Whit

...