trentmkelly/LessWrong-43k huggingface dataset:
Nobody knows how to reliably test for AI safety
by marcusarvan — 2023-03-27T19:48:23.531Z

Large-language AI models such as GPT-3 and 4 do many incredible things—but they also learn to do many unexpected things. Developers at Microsoft were surprised when their Chatbot started threatening people. And recent research has shown that chatbots learn all kinds of other unexpected things all by themselves that researchers never predicted.

This unpredictability has recently led many people to suggest that far more regulation is needed to ensure that AI are developed in a safe way. As Elon Musk put it, “I think we need to regulate AI safety, frankly … It is, I think, actually a bigger risk to society than cars or planes or medicine.” Others have gone further, suggesting that we should slow down AI development altogether.

Who is right?

To answer this question in an informed manner, we need to answer a more basic question: is there any way to reliably test whether AI is safe?

The answer, shockingly, is no.

The problem, right now and for the foreseeable future, is that nobody knows how to determine what AI are actually learning.

As Holden Karnofsky, co-founder and co-CEO of Open Philanthropy, recently stated in an interview:

If you look at this current state of machine learning, it’s just very clear that we have no idea what we’re building …

When Bing chat came out and it started threatening users and, you know, trying to seduce them and god knows what, people asked, why is it doing that? And I would say not only do I not know, but no one knows because the people who designed it don’t know, the people who trained it don’t know.

The reason why developers have no idea what they’re building is simple. Large language models like GPT-4 have well over 500 billion parameters. Developers know that those 500 billion parameters are able to produce convincing human-like language—but they don’t actually know what else the 500 billion parameters are actually learning. There are simply too many of them. The parameters are so complex that they have “No Human Interpretability.”

This is why researchers have been so surprised at what chatbots actually do. No one predicted that Microsoft’s Sydney would start threatening people—or that chatbots would start learning other things on their own—because no one is in a position to understand how all of her 500 billion parameters actually work.

This means that the only way that we c...
New blog: Planned Obsolescence
by Ajeya Cotra — 2023-03-27T19:46:25.429Z
Link: https://www.planned-obsolescence.org/

Kelsey Piper and I just launched a new blog about AI futurism and AI alignment called Planned Obsolescence. If you’re interested, you can check it out

here

.

Both of us have thought a fair bit about what we see as the biggest challenges in technical work and in policy to make AI go well, but a lot of our thinking isn’t written up, or is embedded in long technical reports. This is an effort to make our thinking more accessible. That means it’s mostly aiming at a broader audience than LessWrong and the EA Forum, although some of you might still find some of the posts interesting.

So far we have seven posts:

What we're doing here

"Aligned" shouldn't be a synonym for "good"

Situational awareness

Playing the training game

Training AIs to help us align AIs

Alignment researchers disagree a lot

The ethics of AI red-teaming

Thanks to ilzolende for formatting these posts for publication. Each post has an accompanying audio version generated by a voice synthesis model trained on the author's voice using

Descript Overdub

.

You can submit questions or comments to mailbox@planned-obsolescence.org.
Resources to see how people think/approach mathematics and problem-solving
by zef — 2023-03-27T19:12:44.090Z

I did lots of programming/cybersecurity/other similar flavor things in the past. I'm currently in college and am focusing more on developing my mathematical skillset. I've done linear algebra/multi/group theory, and am currently taking a more advanced class on algebra and rings/fields. In my classes, we do weekly problem-sets and then lectures where we see proofs and go over textbook content. What I feel like I'm lacking in my mathematical learning is exposure to how the experts are thinking about the things we're seeing. What are they tracking or considering? How do they approach the problems and what do they consider?

I usually try to understand what motivates new mathematical object or content, but professors don't focus enough on this. How can I get more of seeing the gears-level model of why things are in a certain way and especially what / how different problems are approached? Do you have recommendations in terms of links/books/articles where I can find more of this kind of perspective?
Staggering Hunters
by Screwtape — 2023-03-27T19:11:50.333Z

Summary: A stag hunt is a situation where people try to coordinate on doing a big, important task but have reasons to individually prefer a safe and comfortable task. Staggering Hunters is a gamified variation of the Stag Hunt.

Tags: Medium, Experimental

Purpose: Practice coordinating with people, even through communication difficulties.

Materials: A bunch of “coins” (which can be small change or poker chips or beads or any other collection of small objects) and a pile of six sided dice (which can be any similar random number generator.)

Announcement Text: The Stag Hunt is a bit of game theory, similar to the more popularly known Prisoner's Dilemma. The idea is that a group of hunters are going out, and they can either cooperate to hunt Stag for a large benefit that only works if everyone chooses it, or individually hunt Rabbit for a small benefit that works even if not everyone agrees. Cooperation is harder than we think; not everybody is always on the same page about what we’re trying to do. Staggering Hunters is a gamified version of this! The LessWrong post here is suggested reading, though you don’t have to have read it to show up and play. We’re going to play the game, then discuss how it went and try again to see if we can do better.

Description: Put the pile of coins and dice in the middle of the table, in reach of as many players as possible.

Explain the following rules:

Definitions: There’s a Pile of Coin and a Pile of dice. Those are like the bank in board games, when we add Coin we can do it from the Pile. The Pot is different; at the end of a round everyone will divide up the Coin in the Pot. We’re going to play for three rounds.
Setup: Everyone rolls a die in secret, then takes Coin from the central pile for what they rolled. (You rolled a 5, take 5 coins.) Keep your current Coin supply secret.
Each round, do the following:

Silence: Roll a die. If it’s 4, 5 or 6 you can talk, if not, stay silent this round. (This includes communication like writing on a notepad, giving a thumbs up, or other deliberate non-verbal communication.)
Discuss: Talk to each other (if you can) about whether you want to hunt stag (pay 5 coins. if successful, add 10 Coins to the Pot) or rabbit (pay 1 coin. If successful, add 3 coins to the Pot) Remember that not all players can speak, even to say that they can’t speak.
Decide: Everyone, speakin

...
Bermudez Interface Problem
by Motor Vehicle — 2023-03-27T18:11:48.133Z

I hope this question makes sense although given my confusion it has come out as word vomit and not a pointed question. So sorry for that.

How does commonsense psychological explanation (folk psychology) interface with the explanations of cognition and mental operations given by scientific psychology, cognitive science, cognitive neuroscience and the other levels in the explanatory hierarchy?

Does comparing neurons firing with beliefs/desires involve a type distinction that renders belief/desires irreducible to patterns of neuron activity? Is one a belief and the other a belief about a belief? Which is the map and which is the terriotory, or are both the map or both the territory? I am entirely lost.
Would you be a better RLHF labeler than GPT-4?
by kache — 2023-03-27T18:10:04.108Z

Have you ever tried to label your own data? Have you ever tried to run an evaluation, yourself, on a system that you've built? It's difficult. We, humans, are imperfect. We are terrible at labeling. We are also expensive, and difficult to scale.
Lets say you wanted to create a high quality instruct dataset, today. What would generate a better output? A high resource model? Or, a farm of hired humans?
How many human RLHF labels are incorrect?
How many benchmarks are our machines now superhuman at?
If we're not bootstrapped, we will be soon.
LLM Powered LW Search
by odraode17 — 2023-03-27T18:09:32.958Z

I find LessWrong Search Function quite weak in returning the result I am looking for, possibly I am not great at selecting keywords, possibly we could use an LLM to help anyone search faster and better match their intent? This is a humble suggestion that will only get traction if the community shares the pain..
P.S.: I could not effectively search if this suggestion was already made in the past..
Announcing the Swiss Existential Risk Initiative (CHERI) 2023 Research Fellowship
by Tobias H — 2023-03-27T16:36:25.508Z

The

Swiss Existential Risk Initiative

(CHERI) is excited to announce the launch of its

2023 Research Fellowship

, an 8-week in-person program focused on global catastrophic risk research.

The fellowship will take place from July 3rd to August 27th, 2023, in the beautiful city of Geneva, Switzerland. This year, we are particularly eager to host fellows interested in exploring governance issues related to global catastrophic risk mitigation, given our close connections with policy organizations in the field. That being said, we also warmly welcome and anticipate having several fellows engaged in more technical research questions.

Apply here

before April 16th, 11:59PM CET.

By participating in the CHERI Research Fellowship, you'll have the opportunity to:

Strengthen your research portfolio and test your fit for research in global catastrophic risks.
Receive mentorship from experienced researchers who will provide valuable support and guidance.
Network and collaborate with like-minded individuals and professionals in the field.
Develop essential skills and knowledge to shape the future of global catastrophic risk research.

In addition to these benefits, the Research Fellowship offers:

A $5,500 stipend, free accommodation, and travel expense support.
Co-working in a shared office at a policy hub in Geneva with access to the UN library.
Learning opportunities through talks and workshops from various GCR researchers from all over Europe.
A stimulating and socially enriching summer experience amidst the breathtaking Swiss landscape.

The 2022 Cohort

Who are we looking for?

We are seeking proactive and motivated individuals from a diverse range of subject areas who have the potential to make a significant impact in the field of global catastrophic risk research. Ideal candidates will possess strong analytical and critical thinking skills, a genuine interest in understanding and mitigating existential risks, and a commitment to making a positive difference in the world.

Eligibility criteria:

Applicants must be currently pursuing a Bachelor, Master, or PhD (in any discipline), starting their studies in 2023, graduating in 2023, or have graduated in the past year.
Willing and able to spend July & August 2023 in Geneva, Switzerland (we can help with short-stay Schengen visas).
Available to do research full-time over the 8-week period.

Application process

The first stag...
Industrialization/Computerization Analogies
by Gordon Seidoh Worley — 2023-03-27T16:34:21.659Z

I have two motivations in this post:

thanks to GPT-4, lots of folks looking at automation via AI seriously for the first time and are unfamiliar with much of the thinking and writing about transformative AI from the past 20-30 years
I think the standard analogy of AI to steam power is slightly off, and I want to make at least the outline of the case for it, even if I don't have time to invest in making the full case

Thus I'm going to highlight what I think are important analogies between industrialization and computerization and what I think that can tell us to expect from transformative AI in some ways. I want to be clear this is not very careful work so there's lots of caveats and exceptions and such. My aim is to 80/20 the analogy and the explanation.

steam engine : industrialization :: computer : computerization

the computer is roughly the equivalent of the steam engine for the computer age (hence the name)
like the steam engine, initially it had limited applications and minimal impact on the economy
then, after a few decades, improvements in design enabled the automation of key work (for the steam engine, textiles, for computers, data processing)
over the next few decades, design improvements enabled more areas of the economy to be automated, making individual workers more productive and making us all richer
then a big new technology transformed the world

assembly line/mass production :: transformative AI

the assembly line and the creation of the modern factory transformed the world and made us all a lot richer
prior to the assembly line, we can think of steam engines mostly automating jobs that used to be done by hand

those jobs were still mostly done under the old paradigm, but in an a more automated way

the genius of the assembly line and the mass production it enabled was to reframe how stuff got made in ways that enabled a higher degree of productivity
transformative AI, of which GPT-4 is the first instance (sort of like Ford's Highland Park), is computers having the sort of impact as the assembly line did, dramatically reconfiguring work rather than simply automating old ways of doing it

implications for jobs and productivity

we should expect transformative AI to make us all a lot richer quickly, much like mass production did

not everyone did/will like this, because richer mostly comes by making previously expensive things cheap

old ways of working will be outcompet

...
Lessons from Convergent Evolution for AI Alignment
by Jan_Kulveit — 2023-03-27T16:25:13.571Z

Prelude: sharks, aliens, and AI

If you go back far enough, the ancestors of sharks and dolphins look really different:

An

acanthodian

, ancestor to modern sharks[1]

A

pakicetus

, ancestor to modern dolphins[2]

But modern day sharks and dolphins have very similar body shapes:

Bodies of a shark, ichthyosaurus and dolphin. Generated in Midjourney.

This is a case of convergent evolution: the process by which organisms with different origins develop similar features. Both sharks and dolphins needed speed and energy efficiency when moving in an environment governed by the laws of hydrodynamics, and so they converged on a pretty similar body shape.

For us, this isn’t very surprising, and doesn’t require much knowledge of evolution: we have a good intuitive understanding of how water works, and humans knew a lot of the underlying maths for the laws of hydrodynamics before they understood anything about evolution. Starting from these laws, it isn’t very surprising that sharks and dolphins ended up looking similar.

But what if instead of starting with knowledge of hydrodynamics and then using that to explain the body shape of sharks and dolphins, we started with only knowledge of sharks’ and dolphins’ body shape, and tried to use that to explain underlying laws?

Let’s pretend we’re alien scientists from an alternative universe, and for some weird reason we only have access to simplified 3D digital models of animals and some evolutionary history, but nothing about the laws of physics in the human/shark/dolphin universe. My guess is that these alien scientists would probably be able to uncover a decent amount of physics and a fair bit about the earth’s environment, just by looking at cases of convergent evolution.

If I’m right about this guess, then this could be pretty good news for alignment research. When it comes to thinking about AI, we’re much closer to the epistemic position of the alien scientist: we either don't know the ‘physics’ of life and intelligence at all, or are only just in the process of uncovering it.

But cases of convergent evolution might help us to deduce deep selection pressures which apply to AI systems as well as biological ones. And if they do, we might be able to say more about what future AI systems might look like, or, if we are lucky, even use some of the selection pressures to shape what systems we get.

Introduction

This post...
GPT-4 is bad at strategic thinking
by Christopher King — 2023-03-27T15:11:47.448Z

GPT-4 is known to pretty good at chess (see I played chess against ChatGPT-4 and lost! for one example). However, GPT-4 does not seem to be very good at strategic reasoning in general (it only really can do it if there is a greedy search algorithm).

I tried Hex and Connect4, it failed at both despite being able to explain the rules and even display the board with ASCII art. I was wondering if maybe it just has bad spatial reasoning, so I tried puzzles in natural language based on logical constraints. It failed these as well unless they were quite simple.

I even made a variant of chess up on the spot where the goal is to get any piece to the bank rank instead of capturing the King. It didn't stop me from "sacking" my queen by moving it to the bank rank as soon as their was a gap. So if it has an internal model of chess, it didn't figure out how to apply it to new objectives.

So I think GPT-4 must've learned a rudimentary chess engine; it is not applying general strategic reasoning to chess.

This doesn't necessarily mean GPT-4 can't be agentic, but it does suggest it is either a narrow one or a dumb one (or it's hiding its abilities).
The salt in pasta water fallacy
by Thomas Sepulchre — 2023-03-27T14:53:07.718Z

There is a particular fallacy I encounter often, and I don't know if it has a name already. I call it the salt in pasta water fallacy. In this post I will present the fallacy, discuss why I believe it is a fallacy, and give a concrete example of it happening.

Salt in pasta water

The archetypical example of the fallacy is the following claim: Adding salt to pasta water reduces cooking time.

The proof for this statement can be described as follows:

When cooking pasta, the higher the temperature of the water, the lower the cooking time
The temperature of the water is its boiling point
Adding salt to water increases its boiling point
(3+2) Adding salt to water increases the temperature
(4+1) Adding salt to water decreases the cooking time

"All of those points seem very reasonable", you may be wondering, "where is the fallacy?" Is there some sneaky chemistry phenomenon going on?

AFAIK no, there is none. My objection goes a bit deeper.

Let's do the math

First of all, can we estimate how big the effect is? To do this, we need to know three things:

How does the temperature affect cooking time? I found a paper[1] from which I derive the following first-order approximation: each extra °C decreases the cooking time of 30 seconds.
How does salt affect the temperature? I found another paper[2] which claims that the boiling point is raised one degree for every mole of salt (i.e. for every 58g) per liter.
How much salt does one typically add to pasta water? This is more personal, but it seems that about 5g/l is standard

So, now we simply multiply all of this and obtain... 2.6s. By adding salt to pasta water, one saves 2.6 seconds of cooking time.

Where is the fallacy again?

You may be wondering where I am going. The initial statement seemed true, I just added numbers, so what? Where is the fallacy.

The fallacy, I claim, is that 2.6s are nothing. Adding salt does not decrease cooking time. It doesn't increase it either, it just does nothing (wrt cooking time at least). This may seem weird to you, and I will try to detail it a bit more. The fallacy, I believe, lies in some play-on-word.

Assume for example that my friend Bob and I are in front of a cooking pot, waiting for the pasta to be ready. And Bob says "adding salt will reduce cooking time". What does this sentence mean?

Does it only mean that adding salt will reduce cooking time?
Does it mean that we should add salt, b

...
CAIS-inspired approach towards safer and more interpretable AGIs
by Peter Hroššo — 2023-03-27T14:36:12.712Z

Epistemic status: a rough sketch of an idea

Current LLMs are huge and opaque. Our interpretability techniques are not adequate. Current LLMs are not likely to run hidden dangerous optimization processes. But larger ones may.

Let's cap the model size at the currently biggest models, ban everything above. Let's not build superhuman level LLMs. Let's build human level specialist LLMs and allow them to communicate with each other via natural language. Natural language is more interpretable than the inner processes of large transformers. Together, the specialized LLMs will form a meta-organism which may become superhuman, but it will be more interpretable and corrigible, as we'll be able to intervene on the messages between them.

Of course, model parameter efficiency may increase in the future (as it happened with Chinchilla) -> we should monitor this and potentially lower the cap. On the other hand, our mechanistic interpretability techniques may improve, so we may increase the cap, if we are confident it won't do harm.
This idea seems almost trivial to me, but I haven't seen it discussed anywhere, so I'm posting it early to gather feedback why this might not work.
An Overview of Sparks of Artificial General Intelligence: Early experiments with GPT-4
by Annapurna — 2023-03-27T13:44:43.805Z
Link: https://jorgevelez.substack.com/p/gpt4

Analyzing one of the most fascinating papers on Artificial Intelligence today

I’ve been wanting to write about the advancements of Artificial Intelligence (AI) since I started using chatGPT back in December. The developments and advancement of AI in the past few months are probably the most exciting technological breakthroughs in my lifetime. Several people,

including Bill Gates

, seem to agree with me.

I have been very brain scattered on the subject, simply because it is so mind blowing. It also feels like it is advancing very fast. So instead of providing my not-so-educated thoughts on the subject, I decided to review and simplify a

preprint released by Microsoft Research back on March 23.

Overview

Over the last few years, the advancements in Artificial Intelligence have been focused on

Large Language Models

(LLMs), which are neural networks that are trained to predict the next word in a sentence. The latest versions of these models, including

GPT-4

(the model this paper is focused on) are showing that its capabilities go beyond just predicting the next word. The latest LLMs seem to demonstrate almost complete knowledge of the general world as well as understanding the syntax and semantics of most languages humans use. In this paper the team at Microsoft Research demonstrates the novel capabilities of GPT-4 as well as its limitations.

Approach

The team took the following approach to analyze the capabilities of GPT-4:

We aim to generate novel and difficult tasks and questions that convincingly demonstrate that GPT-4 goes far beyond memorization, and that it has a deep and flexible understanding of concepts, skills, and domains (a somewhat similar approach was also proposed in [CWF+22]). We also aim to probe GPT-4’s responses and behaviors, to verify its consistency, coherence, and correctness, and to uncover its limitations and biases. We acknowledge that this approach is somewhat subjective and informal, and that it may not satisfy the rigorous standards of scientific evaluation. However, we believe that it is a useful and necessary first step to appreciate the remarkable capabilities and challenges of GPT-4, and that such a first step opens up new opportunities for developing more formal and comprehensive methods for testing and analyzing AI systems with more general intelligence.

The team wanted to see if the model could reason, plan, solve problems, learn both q...