trentmkelly/LessWrong-43k huggingface dataset:
AI #7: Free Agency
by Zvi — 2023-04-13T16:20:02.853Z

This week’s biggest AI development was the rise of AutoGPT and the concept of turning GPTs into agents via code wrapping that gives them memory, plans and the ability to spin up instantiations. I moved that to its own post, which I would prioritize over most sections of this one.

There are also two other things that got cut out and put into the draft pile.

I took out the section ‘Tyler Cowen Suggests America First Strategy and So Much More’ while I figure out how to usefully move such interactions forward. I’m also working on an ‘effort post’ trying to ‘model AI risk’ per his request, I agree with Richard Ngo this is a largely wrong request but it could also be productive.

I also took out a section called ‘Yes, AI Systems Could Actually Kill People’ until I have time to expand and rework it properly as its own post, and incorporate the related poll I ran on Twitter, and also perhaps until I finish reading Daemon since I was told I was cribbing its plot.

There’s still plenty to talk about, although it was what passes these days for a quiet week. I am already shifting more effort towards more focused questions.

Table of Contents

Introduction. The inevitable rise of the AI agent, covered in another post.

Table of Contents. See table of contents.

Language Models Offer Mundane Utility. Remarkably little new utility.

Fun With Image Generation. ControlNet 1.1 is out, plug-in interfaces are an option I guess, and Washington Post warns of AI porn.

They Took Our Jobs. Copywriting is falling fast, Americans continue to be unusually pessimistic about AI.

GPT-4 Fails an Economics Test. Yes, it can get by Bryan Caplan, yet it is no match for Steve Landsburg and his request that students actually think. For now.

In Other AI News. Bug bounties launched, prizes awarded, warnings issued, political lobbying enabled, new chatbot to try out.

Quiet Speculations. Taleb being true to himself, Ajeya Corta and Jack Clark suggesting AI already accelerating AI, the cost of an Uber ride when self-driving cars are coming, getting the world domination plans out of the training data.

Deepfaketown and Botpocalypse Soon. Criminals attempt a fake kidnapping using a deepfaked voice, fail to do their homework, get foiled within four minutes.

Eliezer Yudkowsky on The Lunar Society Podcast. Each EY podcast is very diff

...
Navigating the Open-Source AI Landscape: Data, Funding, and Safety
by André Ferretti — 2023-04-13T15:29:05.234Z
Link: https://forum.effectivealtruism.org/posts/N25EARxvbxYJa5pbB

Introduction

Open-source AI promises to democratize technology, but it can also be abused and may lead to hard-to-control AI. In this post, we'll explore the data, ethics, and funding behind these models to discover how to balance innovation and safety.

Summary

Open-source models, like LLaMA and GPT-NeoX, are trained on huge public datasets of internet data, such as

the Pile

, which has 800 GB of books, medical research, and even emails of Enron employees before their company went bankrupt and they switched careers to professional hide-and-seek.

After the unexpected leak of Meta’s LLaMA, researchers cleverly enhanced it with ChatGPT outputs, creating chatbots

Alpaca

and

Vicuna

. These new bots perform nearly as well as GPT-3.5 and cost less to train — Alpaca took just 3 hours and

$600

. The race is on to run AI models on everday devices like

smartphones

, even on

calculators

.

The leading image generation model, Stable Diffusion, is developed by Stability AI — a startup that has amassed $100 million in funding, much like Hugging Face (known as the "Github of machine learning"). These two unicorn startups financed a nonprofit to collect 5 billion images for training the model. Sourced from the depths of the internet, this public dataset raises concerns about copyright and privacy, as it includes

thousands

of private medical files.

The open-source AI community wants to make AI accessible and prevent Big Tech from controlling it. However, risks like malicious use of Stable Diffusion exist, as its safety filters can be

easily

removed. Misuse includes virtually

undressing

people. If we're still struggling with how to stop a superintelligent AI that doesn't want to be turned off — like Skynet in Terminator — how can we keep open-source AI from running amok in the digital wild?

Open-source code helps create advanced AI faster by letting people use each other's work, but this could be risky if this AI isn't human-friendly. However, to ensure safety, researchers need access to the AI's "brain," which is why EleutherAI openly shares their models. EleutherAI is especially worried about

catastrophic risks

from AI, with its co-founders even launching Conjecture to focus on AI alignment. They also signed an

open letter

calling for a six-month pause in AI development. But others, like LAION, want to push forward and support ...
On AutoGPT
by Zvi — 2023-04-13T12:30:01.059Z

The primary talk of the AI world recently is about AI agents (whether or not it includes the question of whether we can’t help but notice we are all going to die.)

The trigger for this was AutoGPT, now number one on GitHub, which allows you to turn GPT-4 (or GPT-3.5 for us clowns without proper access) into a prototype version of a self-directed agent.

We also have a paper out this week where a simple virtual world was created, populated by LLMs that were wrapped in code designed to make them simple agents, and then several days of activity were simulated, during which the AI inhabitants interacted, formed and executed plans, and it all seemed like the beginnings of a living and dynamic world. Game version hopefully coming soon.

How should we think about this? How worried should we be?

The Basics

I’ll reiterate the basics of what AutoGPT is, for those who need that, others can skip ahead. I talked briefly about this in AI#6 under the heading ‘Your AI Not an Agent? There, I Fixed It.’

AutoGPT was created by game designer Toran Bruce Richards.

I previously incorrectly understood it as having been created by a non-coding VC over the course of a few days. The VC instead coded the similar program BabyGPT, by having the idea for how to turn GPT-4 into an agent. The VC had GPT-4 write the code to make this happen, and also ‘write the paper’ associated with it.

The concept works like this:

AutoGPT uses GPT-4 to generate, prioritize and execute tasks, using plug-ins for internet browsing and other access. It uses outside memory to keep track of what it is doing and provide context, which lets it evaluate its situation, generate new tasks or self-correct, and add new tasks to the queue, which it then prioritizes.

This quickly rose to become #1 on GitHub and get lots of people super excited. People are excited, people are building it tools, there is a bitcoin wallet interaction available if you never liked your bitcoins. AI agents offer very obvious promise, both in terms of mundane utility via being able to create and execute multi-step plans to do your market research and anything else you might want, and in terms of potentially being a path to AGI and getting us all killed, either with GPT-4 or a future model.

As with all such new developments, we have people saying it was inevitable and they knew it would happen all along, and others that are surprised. We...
Identifying semantic neurons, mechanistic circuits & interpretability web apps
by Esben Kran — 2023-04-13T11:59:51.629Z

15 research projects

on interpretability were submitted to the

mechanistic interpretability Alignment Jam

in January hosted with Neel Nanda. Here, we share the top projects and results. In summary:

Activation patching works on singular neurons, token vector and neuron output weights can be compared, and a high mutual congruence between the two indicates a mono-semantic MLP neuron.
The Automatic Circuit Identification tool (

ACDC

) can be used to infer a circuit for gendered pronouns, and some of these circuits can perform even better than the full model. Hyperparameter tuning for ACDC is also very important.
A three-stage method can be used to automatically identify semantically coherent neurons and describe in human-understandable language what that neuron activates to.
Several tokens that complete the same task with GPT-2 prompts can be replaced with token coordinates between their positions that accomplish the same task with acceptable performance.
Other projects include comparisons of compiled and trained Transformers, web apps for mechanistic interpretability, latent knowledge regularizers, and embedding attention heads.

Join the

interpretability hackathon 2.0

happening this weekend.

We Found " an" Neuron

By Joseph Miller and Clement Neo

Abstract (from the subsequent LessWrong post): We started out with the question: How does GPT-2 know when to use the word "an" over "a"? The choice depends on whether the word that comes after starts with a vowel or not, but GPT-2 can only output one word at a time. We still don’t have a full answer, but we did find a single MLP neuron in GPT-2 Large that is crucial for predicting the token " an". And we also found that the weights of this neuron correspond with the embedding of the " an" token, which led us to find other neurons that predict a specific token.

First, they use the

logit lens

to identify a multi-layer perceptron (MLP) layer in the Transformer where the difference between predicting " an" and " a" is the largest (logit is a way of representing the model's probability for what the next token should be).

They then use activation patching

(Meng et al, 2022)

to see how specific layers in the model contribute to the prediction of " an". In activation patching, you save the activations on a prompt such as "I climbed the apple tree and picked[...]" and replace these saved activations in a run with&nbs...
Trying AgentGPT, an AutoGPT variant
by Gunnar_Zarncke — 2023-04-13T10:13:41.316Z

I saw this announcement on Twitter:

https://twitter.com/asimdotshrestha/status/1644883727707959296

I gave it a trial run and asked it to Solve the hard AI Alignment problem:

At this point the trial had reached 30 seconds and expired.

I think it is funny that it "thinks" the task executed successfully. I guess that's because in most texts ChatGPT has read how researchers describe their successes but not the actual work that is backing it up. I guess the connection to the real world is what will throw off such systems until they are trained on more real-world-like data.
Announcing Epoch’s dashboard of key trends and figures in Machine Learning
by Jsevillamol — 2023-04-13T07:33:06.936Z
Link: https://epochai.org/trends

Developments in Machine Learning have been happening extraordinarily fast, and as their impacts become increasingly visible, it becomes ever more important to develop a quantitative understanding of these changes. However, relevant data has thus far been scattered across multiple papers, has required expertise to gather accurately, or has been otherwise hard to obtain.

Given this, Epoch is thrilled to announce the launch of our new dashboard, which covers key numbers and figures from our research to help understand the present and future of Machine Learning. This includes:

Training compute requirements
Model size, measured by the number of trainable parameters
The availability and use of data for training
Trends in hardware efficiency
Algorithmic improvements for achieving better performance with fewer resources
The growth of investment in training runs over time

Our dashboard gathers all of this information in a single, accessible place. The numbers and figures are accompanied by further information such as confidence intervals, labels representing our degree of uncertainty in the results, and links to relevant research papers. These details are especially useful to illustrate which areas may require further investigation, and how much you should trust our findings.

Beyond accessibility, bringing these figures together allows us to compare and contrast trends and drivers of progress. For example, we can verify that growth in training compute is driven by improvements to hardware performance and rising investments:

We can also see that performance improvements have historically been driven by algorithmic progress and training compute growth by comparable amounts:

Overall, we hope that our dashboard will serve as a valuable resource for researchers, policymakers, and anyone interested in the future of Machine Learning.

We plan on keeping our dashboard regularly updated, so stay tuned! If you spot an error or would like to provide feedback, please feel free to reach out to us at info@epochai.org.

Visit now the dashboard
What is the best source to explain short AI timelines to a skeptical person?
by trevor — 2023-04-13T04:29:03.166Z

This is for a person with no ML background. He is 55 years old, he liked the sequences and I recently managed to convince him that AI risk is serious by recommending a bunch of Lesswrong posts on it, but he still thinks it's astronomically unlikely that AGI is <80 years away.

There are a lot of other people like this, so I think it's valuable to know what the best explainer is, more than just in my case.
"Aligned" foundation models don't imply aligned systems
by Max H — 2023-04-13T04:13:49.984Z

[The basic idea in this post is probably not original to me, since it's somewhat obvious when stated directly. But it seems particularly relevant and worth distilling in light of recent developments with LLM-based systems, and because I keep seeing arguments which seem confused about it.]

Alignment is a property of agents, or even more generally, of systems. An "aligned model" is usually[1] a type error.

Often when a new state-of-the-art ML model is developed, the first thing people ask is what it can do when instantiated in the most obvious way possible: given an input, execute the function represented by the model on that input and return the output. I'll refer to this embodiment as the "trivial system" for a given model.

For an LLM, the trivial system generates a single token for a given prompt. There's an obvious extension to this system, which is to feed the predicted token and the initial prompt back into the model again, and repeat until you hit a stop sequence or a maximum length. This is the system you get when you make a single call to the OpenAI text or chat completion API. I'll name this embodiment the "trivial++ system".

You can take this much further, by building a chatbot interface around this API, hooking it up to the internet, running it in an agentic loop, or even more exotic arrangements of your own design. These systems have suddenly started working much better with the release of GPT-4, and AI capabilities researchers are just getting started. The capabilities of any particular system will depend on both the underlying model and the ways it is embodied: concretely, you can improve Auto-GPT by: (a) backing it with a better foundation model, (b) giving it access to more APIs and tools, and (c) improving the code, prompting, and infrastructure that it runs on.

Perhaps models to which techniques like RLHF have been applied will make it easy to build aligned systems and hard or impossible to build unaligned systems, but this is far from given.

Things like RLHF and instruction prompt tuning definitely make it easier to build powerful systems out of the foundation models to which they are applied. Does an RLHF'd model make it easier to build Order-GPT and harder to build Chaos-GPT? (I'm not sure; no Auto-GPT applications seem to really be working particularly well yet, but I wouldn't count on that trend continuing.)

So while RLHF is d...
Using ChatGPT for memory reconsolidation?
by warrenjordan — 2023-04-13T01:27:46.331Z

I just tried a IFS bot based on GPT3.5 and I actually enjoyed it! Didn't make a breakthrough as I've done it on my own in the past, but definitely felt similar to my experiences.

Another use case I can think of is using ChatGPT to help me with memory reconsolidation. I had a breakthrough with this memory reconsolidation that helped resolve my childhood trauma of my father passing away. Wondering if there's some prompting I could leverage to help me do more memory reconsolidation for some other lingering issues I have.

Of course, nothing replaces real therapy but my issues aren't worth spending money at this point.
Independence Dividends
by jefftk — 2023-04-13T01:20:01.807Z

Our oldest is in third grade, and sings in the school
chorus. It's after school on Wednesdays, which is fine. Except on
half-day Wednesdays it's before school, which is less fine.
Instead of needing to be to school at 8am she needs to be there at
7:15. She really likes chorus, and feels strongly that it's worth
waking up for, but as the one getting up early with her, the tradeoff
feels a bit different to me.

This morning, for the first time, when my alarm went off she
was already at school. We had talked about how I thought she was
ready to do this, and how I really didn't want to be waking up early
when she didn't need me, and we agreed she would handle it. Which she
did well! When I saw her briefly at school, dropping the middle
child, she was very proud of herself.

Cultivating independence can be
more work in the moment, but it's really great when the
investment pays dividends.
AI x-risk, approximately ordered by embarrassment
by Alex Lawsen  — 2023-04-12T23:01:00.561Z

Advanced AI systems could lead to existential risks via several different pathways, some of which may not fit neatly into traditional risk forecasts. Many previous forecasts, for example the well known report by Joe Carlsmith, decompose a failure story into a conjunction of different claims, and in doing so risk missing some important dangers. ‘Safety’ and ‘Alignment’ are both now used by labs to refer to things which seem far enough from existential risk reduction that using the term ‘AI notkilleveryoneism’ instead is becoming increasingly popular among AI researchers who are particularly focused on existential risk.

This post presents a series of scenarios that we must avoid, ranked by how embarrassing it would be if we failed to prevent them. Embarrassment here is clearly subjective, and somewhat unserious given the stakes, but I think it gestures reasonably well at a cluster of ideas which are important, and often missed by the kind of analysis which proceeds via weighing the incentives of multiple actors:

Sometimes, easy problems still don’t get solved on the first try.
An idea being obvious to nearly everyone does not mean nobody will miss it.
When one person making a mistake is sufficient for a catastrophe, the relevant question is not whether the mistake will be obvious on average, but instead whether it will be obvious to everyone with the capacity to make it.

The scenarios below are neither mutually exclusive nor collectively exhaustive, though I’m trying to cover the majority of scenarios which are directly tackled by making AI more likely to try to do what we want (and not do what we don’t). I’ve decided to include some kinds of misuse risk, despite this more typically being separated from misalignment risk, because in the current foundation model paradigm there is a clear way in which the developers of such models can directly reduce misuse risk via alignment research.

Many of the risks below interact with each other in ways which are difficult to fully decompose, but my guess is that useful research directions will map relatively well onto reducing risk in at least one of the concrete scenarios below. I think people working on alignment might well want to have some scenario in mind for exactly what they are trying to prevent, and that this decomposition might also prove somewhat useful for risk modelling. I expect that some criticism o...
AXRP Episode 20 - ‘Reform’ AI Alignment with Scott Aaronson
by DanielFilan — 2023-04-12T21:30:06.929Z

YouTube link

How should we scientifically think about the impact of AI on human civilization, and whether or not it will doom us all? In this episode, I speak with Scott Aaronson about his views on how to make progress in AI alignment, as well as his work on watermarking the output of language models, and how he moved from a background in quantum complexity theory to working on AI.

Topics we discuss:

‘Reform’ AI alignment

Epistemology of AI risk

Immediate problems and existential risk

Aligning deceitful AI

Stories of AI doom

Language models

Democratic governance of AI

What would change Scott’s mind

Watermarking language model outputs

Watermark key secrecy and backdoor insertion

Scott’s transition to AI research

Theoretical computer science and AI alignment

AI alignment and formalizing philosophy

How Scott finds AI research

Following Scott’s research

Daniel Filan:
Hello, everyone. In this episode, I’ll be speaking with Scott Aaronson. Scott is a professor of computer science at UT Austin and he’s currently spending a year as a visiting scientist at OpenAI working on the theoretical foundations of AI safety. We’ll be talking about his view of the field, as well as the work he’s doing at OpenAI. For links to what we’re discussing, you can just check the description of this episode and you can read the transcript at axrp.net. Scott, welcome to AXRP.

Scott Aaronson:
Thank you. Good to be here.

‘Reform’ AI alignment

Epistemology of AI risk

Daniel Filan:
So you recently wrote this blog post about something you called reform AI alignment: basically your take on AI alignment that’s somewhat different from what you see as a traditional view or something. Can you tell me a little bit about, do you see AI causing or being involved in a really important way in existential risk anytime soon, and if so, how?

Scott Aaronson:
Well, I guess it depends what you mean by soon. I am not a very good prognosticator. I feel like even in quantum computing theory, which is this tiny little part of the intellectual world where I’ve spent 25 years of my life, I can’t predict very well what’s going to be discovered a few years from now in that, and if I can’t even do that, then how much less can I predict what impacts AI is going to have on human civilization over the next...