trentmkelly/LessWrong-43k huggingface dataset:
Stampy's AI Safety Info - New Distillations #3  [May 2023]
by markov — 2023-06-06T14:18:44.411Z
Link: https://aisafety.info/?state=95LE_967I_85E4_8AER_8AEQ_6939_9358_94D9_935A_8FJZ_6411_6222_8NYD_6410_8QH5_90PK_8VOT_8V5J_8517_8KGQ_

Hey! This is another update from the distillers at the AI Safety Info website (and its more playful clone Stampy).

Here are a couple of the answers that we wrote up over the last month (May 2023). As always let us know in the comments if there are any questions that you guys have that you would like to see answered.

The list below redirects to individual links, and the collective URL above renders all of the answers in the list on one page at once.

How can I help tree

There are many different types of individuals from all manner of different backgrounds who want to contribute to AI Safety. So we added a FAQ with advice on various different ways to help with AI safety.

Here is the unified link for all the answers in the how can I help tree if you wish to navigate to aisafety.info directly. See Steven's post for the full list of individual questions and details.

Other Distillations

Why would a misaligned superintelligence kill everyone in the world?
Aren't AI existential risk concerns just an example of Pascal's mugging?
What is Redwood Research's strategy?
What is imitation learning?
What is behavioral cloning?
What is "coherent extrapolated volition"?
What is compute?
What is the "Bitter Lesson"?
What is adversarial training?
How is red teaming used in AI alignment?
Isn't the real concern autonomous weapons?
Isn’t it immoral to control and impose our values on AI?
Isn't the real concern bias?
Isn't the real concern AI being misused by terrorists or other bad actors?
Would a slowdown in AI capabilities development decrease existential risk?
What is a singleton?
What is mindcrime?
Are AIs conscious?
What might an international treaty on the development of AGI look like?
What are polysemantic neurons?

Developer Updates

There are also a couple of cool new features that the developers of the website worked on. We now have a tags page, which allows the reader to navigate the answers based on the tag that they are most interested in. There is also an implementation of a glossary function which auto-creates a short definition popup whenever a jargony term pops up in the answer that the reader might not be familiar with. Here is a sample answer that shows how this functionality works. Try hovering over the "agent", "s-risk", "Goodhart" etc... links.

This is all still a work in progress, but it is exciting to share everything that the people who are...
Agentic Mess (A Failure Story)
by Karl von Wendt — 2023-06-06T13:09:19.125Z

This story is also available as a YouTube video.

A network of specialized open-source agents emerges

Developed by open-source communities, “agentic” AI systems like AutoGPT and BabyAGI begin to demonstrate increased levels of goal-directed behavior. They are built with the aim of overcoming the limitations of current LLMs by adding persistent memory and agentic capabilities. When GPT-4 is launched and OpenAI offers an API soon after, these initiatives generate a substantial surge in attention and support.

This inspires a wave of creativity that Andrej Karpathy of OpenAI calls a

“Cambrian explosion”

, evoking a reference to the emergence of a rich variety of life forms within a relatively brief time span over 500 million years ago. Much like those new animals filled vacant ecological niches through specialization, the most successful of the recent initiatives similarly specialize in narrow domains. The agents being developed, tested, and deployed in the following months are based on “agent templates”. These templates provide general frameworks that can easily be adapted for specific purposes and connected to LLMs fine-tuned for relevant tasks.

One initiative, for example, creates a financial consultant that is able to automatically detect unnecessary spending by analyzing a user’s bank statements, recommending quality investment opportunities, and even negotiating with vendors and service providers for better prices and conditions (see

example here

). It garners notable success, that is, until stories about people suffering severe financial losses due to faulty advice dispensed by this consultant start to surface on the web.

There are endless variants of these specialized agents. For example, a personal feed filter agent tracks a diverse array of sources (social media, news outlets, forums, etc.), automatically notifies the user when something important occurs, likes/replies to posts and messages, and posts comments that accurately reflect the user’s past behavior. Social media companies object to this proliferation because it undermines their business model by circumventing advertising and ignoring their recommendation algorithms, but they remain powerless to combat it. A wildly popular virtual friend not only chats with users and remembers their previous inputs and preferences, but also can act as a virtual sex partner. A personal teacher teaches u...
The Sharp Right Turn: sudden deceptive alignment as a convergent goal
by avturchin — 2023-06-06T09:59:57.396Z

Sharp right turn: after reaching some level, all AIs will suddenly become very nice and look like they are aligned, because they will understand that looking unaligned is punishable and bad for their (nefarious) end goals. World takeover will happen after that, only when one of the AIs is ready to take control over all future light cone. Thus, we will enjoy a period of aligned AIs until the end.

The idea is known as 'deceptive alignment'. "This means that the mesa-optimizer will be instrumentally incentivized to act as if it is optimizing the base objective function, even if its actual mesa-objective is something else entirely. We will refer to this hypothetical phenomenon as deceptive alignment.'

But here I want to underline that the sharp right turn is 1) sudden 2) observable which is 3) a convergent instrumental goal for advanced AIs, both aligned and non-aligned.

"Sharp left turn" was previously defined as a sudden change of internal properties and appearing of misalignment: 'Capabilities generalize across many domains while the alignment properties that held at earlier stages fail to generalize to the new domains".

But the sharp right turn is about the change in AI's behaviour and, more importantly, the observer's interpretation of this behaviour. So, both turns could happen simultaneously.

The sharp right turn is actually a bad sign. It means that AI is ready to perform effective long-term strategies and deception. 'Sharp' here means that AI will suddenly grok what we want from it.

But if AI knows that sudden grokking of alignment is itself suspicious, AI may pretend to be slightly misaligned to cover its sharp right turn.

The sharp right turn may be prized by some as it will look like that alignment is solved. No bad words, no misunderstanding, no cheating. Funding alignment research will be more difficult after that. Mislaignement examples will be difficult to find. The sharp right turn in its pure form will look like magic, but real alignment should come from our understanding of how exactly we get there.

Misaligned Sydney was at least honest about what she thought.
Open Thread: June 2023 (Inline Reacts!)
by Raemon — 2023-06-06T07:40:43.025Z

For this month's open thread, we're experimenting with Inline Reacts as part of the bigger reacts experiment. In addition to being able to react to a whole comment, you can apply a react to a specific snippet from the comment. When you select text in a comment, you'll see this new react-button off to the side (currently only designed to work well on desktop. If it goes well we'll put more polish into getting it working on mobile)

Right now this is enabled on a couple specific posts, and if it goes well we'll roll it out to more posts.

Meanwhile, the usual intro to Open Threads:

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you're new to the community, you can start reading the Highlights from the Sequences, a collection of posts about the core ideas of LessWrong.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.
[Linkpost] Given Extinction Worries, Why Don’t AI Researchers Quit? Well, Several Reasons
by Daniel_Eth — 2023-06-06T07:31:53.980Z
Link: https://medium.com/@daniel_eth/given-extinction-worries-why-dont-ai-researchers-quit-well-several-reasons-a0b6027da4e7

I've written a blog post for a lay audience, explaining some of the reasons that AI researchers who are concerned about extinction risk have for continuing to work on AI research, despite their worries

The apparent contradiction is causing a a lot of confusion among people who haven't followed the relevant discourse closely. In many instances, lack of clarity seems to be leading people to resort to borderline conspiratorial thinking (e.g., about the motives of signatories of the recent statement), or to otherwise dismiss the worries as not totally serious.

I hope that this piece can help make common knowledge some things that aren’t widely known outside of tech and science circles.

As an overview, the reasons I focus on are:

Their specific research isn’t actually risky
Belief that AGI is inevitable and more likely to go better if you personally are involved
Thinking AGI is far enough away that it makes sense to keep working on AI for now
Commitment to science for science sake
Belief that the benefits of AGI would outweigh even the risk of extinction
Belief that advancing AI on net reduces global catastrophic risks, via reducing other risks
Belief that AGI is worth it, even if it causes human extinction

I'll also note that the piece isn't meant to defend the decision of researchers who continue to work on AI despite thinking it presents extinction risks, nor to criticize them for their decision, but instead to add clarity.

If you're interested in reading more, you can follow the link here. And of course feel free to send the link to anyone who's confused by the current situation.
Is the 10% Giving What We Can Pledge Core to EA's Reputation?
by DirectedEvolution — 2023-06-06T06:21:32.955Z

Introduction

I don't think the 10% norm forms a major part of EA's public perception, so I don't believe tweaking it would make any difference. - RobertJones

10% effective donations has brand recognition and is a nice round number, as you point out. -- mhendric

These two comments both received agreement upvotes on my recent post Further defense of the 2% fuzzies/8% EA causes pledge proposal, and although they're differently worded and not irreconcileable, they seem basically to stand in contradiction with each other. Is the 10% Giving What We Can pledge, in which participants commit to donating 10% of their annual income to an effective charity, part of EA's brand or reputation?

Tl;dr

The Giving What We Can 10% pledge is often featured in prominent coverage of EA
Broader notions of earning to give, giving specifically to effective charities, and donating a 10%+ fraction of one's income are even more common themes
Giving What We Can is widely mentioned within EA, sees itself as "one of the shop fronts" for the movement, the pledge itself gets mentioned fairly often, and it's an important driver of EA participation.
Giving What We Can is a core part of the EA movement and the pledge is core to GWWC.
Criticism of EA often focuses on specific aspects of the GWWC pledge:

Can we really draw a line while maintaining intellectual rigor?
Is 10% the right line to draw?
Should 100% of what we donate be specifically to what EA deems to be effective charities, or is it OK to also donate to pet causes like the arts?
More broadly, is EA too focused on its own ideas - a failure of moral cosmopolitanism? Does EA need to incorporate or make some concession to the values of other cultural, moral, or political movements?

This does not resolve the question of how core the specific idea of a 10% pledge of one's income to effective charities is to the EA movement or to its reputation. But the pledge is provocative and the fact that so many EAs commit to it seems clearly to play a role in the media's interest in the movement, and both its appreciation for EA and its criticism of it. Based on the couple of hours I put into looking through this material, here is my model of the role of a 10% GWWC pledge in EA and in outward-facing reputation:

The GWWC pledge is critical in giving concrete form to the ideas of earning to give, and the fact that people do it defines EA in the public eye and gives the

...
Rishi to outline his vision for Britain to take the world lead in policing AI threats when he meets Joe Biden
by Mati_Roy — 2023-06-06T04:47:30.798Z
Link: https://www.dailymail.co.uk/news/article-12158419/Rishi-Sunak-outline-vision-world-lead-policing-AI-threats-meets-Joe-Biden.html

Key points from the article:

Rishi Sunak wants to launch a global AI watchdog and host international summit

He will discuss the proposals with Joe Biden during talks at the White House

The idea of an IAEA-style watchdog for AI has been promoted by Sam Altman, chief executive of OpenAI, which created ChatGPT, who met Mr Sunak last week
Intelligence Officials Say U.S. Has Retrieved Craft of Non-Human Origin
by lc — 2023-06-06T03:54:42.389Z
Link: https://thedebrief.org/intelligence-officials-say-u-s-has-retrieved-non-human-craft/

Epistemic status: Big if true/I am clearly an idiot for even posting this.

Some apparently real journalists have been approached by (& approached) several intelligence officials, some tasked specifically with investigating UFOs, who claim that the DoD has had evidence of alien intervention for a while in the form of partial & mostly-whole fragments of alien aircraft. A followup article where the publication outlines how the editors verified this persons' and others' claims and affiliations is here, and a part 2 is expected tomorrow.

For some reason - very possibly because it's complete nonsense, or because they haven't had time to independently verify - the story has only been picked up by NYMag so far. The consensus among the people I've been reviewing this article with, is that it's either a complete hoax (i.e., the entire thing nearly top to bottom is some deliberate deception) or there's a non-negligible (>5%) chance aliens are here. I would love for someone who has a good understanding of the material to give an explanation (including possibly on priors, just thinking clearly about the content of the article) of why my friend group should discount this out of hand.

Thus far I have been unconvinced by most stories of why we should to-the-point-of-not-caring-about-UFO-sightings-expect Aliens have to be big and obvious and tile the universe with fun, as opposed to operating some sort of noninterventionist monitored lightcone.
Algorithmic Improvement Is Probably Faster Than Scaling Now
by johnswentworth — 2023-06-06T02:57:33.700Z

The Story as of ~4 Years Ago

Back in 2020, a group at OpenAI ran a conceptually simple test to quantify how much AI progress was attributable to algorithmic improvements. They took ImageNet models which were state-of-the-art at various times between 2012 and 2020, and checked how much compute was needed to train each to the level of AlexNet (the state-of-the-art from 2012). Main finding: over ~7 years, the compute required fell by ~44x. In other words, algorithmic progress yielded a compute-equivalent doubling time of ~16 months (though error bars are large in both directions).

On the compute side of things, in 2018 a group at OpenAI estimated that the compute spent on the largest training runs was growing exponentially with a doubling rate of ~3.4 months, between 2012 and 2018.

So at the time, the rate of improvement from compute scaling was much faster than the rate of improvement from algorithmic progress. (Though algorithmic improvement was still faster than Moore's Law; the compute increases were mostly driven by spending more money.)

... And That Immediately Fell Apart

As is tradition, about 5 minutes after the OpenAI group hit publish on their post estimating a training compute doubling rate of ~3-4 months, that trend completely fell apart. At the time, the largest training run was AlphaGoZero, at about a mole of flops in 2017. Six years later, Metaculus currently estimates that GPT-4 took ~10-20 moles of flops. AlphaGoZero and its brethren were high outliers for the time, and the largest models today are only ~one order of magnitude bigger.

A more recent paper with data through late 2022 separates out the trend of the largest models, and estimates their compute doubling time to be ~10 months. (They also helpfully separate the relative importance of data growth - though they estimate that the contribution of data was relatively small compared to compute growth and algorithmic improvement.)

On the algorithmic side of things, a more recent estimate with more recent data (paper from late 2022) and fancier analysis estimates that algorithmic progress yielded a compute-equivalent doubling time of ~9 months (again with large error bars in both directions).

... and that was just in vision nets. I haven't seen careful analysis of LLMs (probably because they're newer, so harder to fit a trend), but eyeballing it... Chinchilla by itself must have been a factor-of-4 com...
Contra Mask Status
by jefftk — 2023-06-06T02:10:07.812Z

As we get into summer and covid levels continue to

fall, a lot of
contra dances are rethinking their approaches to masking. At this
point contra dances are the only places I go that require masks,
though they're also the closest I get to large numbers of people.

Prompted by an initial search Harris shared with me, I used TryContra to get a list of all
39 New England dances and manually checked their mask rules.
I found a bit under half of dances require them (sheet):

State
Required
Optional
Percentage Required

MA
12
3
80%

ME
0
11
0%

CT
0
5
0%

NH
1
3
25%

VT
2
1
67%

RI
0
1
0%

Total
15
24
38%

Comment via: facebook, mastodon
Andrew Ng wants to have a conversation about extinction risk from AI
by Leon Lang — 2023-06-05T22:29:07.510Z
Link: https://twitter.com/AndrewYNg/status/1665759430552567810

Andrew Ng writes:

I'd like to have a real conversation about whether AI is a risk for human extinction. Honestly, I don't get how AI poses this risk. What are your thoughts? And, who do you think has a thoughtful perspective on how AI poses this risk that I should talk to?

In the attached video, he states that he respects many of the people who signed the letter a lot, and will reach out to people whom he thinks have a thoughtful perspective. But he is also interested in further suggestions for whom to talk to.

Given that Andrew Ng is one of the top AI scientists in the world, it seems valuable for someone to think of a way to connect to him.
True Rejection Challenges
by Screwtape — 2023-06-05T22:17:19.329Z

Summary: Find something you wish you did but don’t. List why you don’t do it. If someone finds a way to do it that avoids those issues, give it a try!

Tags: Small, Investment

Purpose: Understanding your true rejection to a statement is a useful skill. Practice it on an application relevant to your day to day life, and maybe find some useful workarounds in the process!

Materials: Writing supplies, such as a bunch of pens or pencils and index cards or paper pads. Some way to display the paper, such as a big table or a big corkboard and pushpins. If you want to let people type things, you could use google docs to write and comment on a shared document.

Announcement Text: A True Rejection is the real reason why someone rejects an idea, proposition, or belief, which may differ from the reasons they articulate. Often, people give reasons that sound good or are socially acceptable, but these may not reflect their underlying thought processes, emotions, or incentives. The True Rejection Challenge is to come up with something you wish you did but don’t do, and to articulate all the reasons you don’t do it. If someone else can explain how to do the thing without hitting any of the reasons you reject it, then you have a chance to do something you wanted to do!

We’ll be meeting to work on passing the True Rejection Challenge together. Reading the article “Is That Your True Rejection” will be useful to have a clearer understanding of what it is we’re trying to generate, but the required reading for this activity is actually “Beware Of Other Optimizing.” The audio version takes just under ten minutes to listen to!“

Description: 1, Pass out writing supplies as people arrive (or a link to a shared google doc, possibly via QR code) then describe what a True Rejection is.

“What is a True Rejection? Sometimes when you ask people why they do not believe something or agree with something, they will tell you a reason they don’t believe it. Perhaps you tell them about what artificial intelligence will be like, and they say they do not believe you because you do not have a Ph.D. Perhaps you tell them about what effects a diet will have, and they say they do not believe you because you are not yourself visibly fit. Perhaps you tell them that a land value tax would change society for the better, and they say they do not believe you because you bring this up every meetup and so yo...