trentmkelly/LessWrong-43k huggingface dataset:
What should AI safety be trying to achieve?
by EuanMcLean — 2024-05-23T11:17:49.204Z

This is the second of three posts summarizing what I learned when I interviewed 17 AI safety experts about their "big picture" of the existential AI risk landscape: how will artificial general intelligence (AGI) play out, how things might go wrong, and what the AI safety community should be doing. See here for a list of the participants and the standardized list of questions I asked.

This post summarizes the responses I received from asking “conditional on ‘AI safety’ preventing a catastrophe, what did AI safety do to prevent such a catastrophe?” and “what research direction will reduce existential risk the most?”

A rough decompositions of the main themes brought up. The figures omit some less popular themes, and double-count respondents who brought up more than one theme.

When asked how AI safety might prevent disaster, respondents focussed most on 1) the technical solutions we might come up with, 2) spreading a safety mindset through AI research, 3) promoting sensible AI regulation, and 4) building a fundamental science of AI. The research directions people were most excited about were mechanistic interpretability, black box evaluations, and governance research.

How to read this post

This is not a scientific analysis of a systematic survey of a representative sample of individuals, but my qualitative interpretation of responses from a loose collection of semi-structured interviews. Take everything here appropriately lightly.

Results are often reported in the form “N respondents held view X”. This does not imply that “17-N respondents disagree with view X”, since not all topics, themes and potential views were addressed in every interview. What “N respondents held view X” tells us is that at least N respondents hold X, and consider the theme of X important enough to bring up.

How could AI safety prevent catastrophe?

Q3 Imagine a world where, absent any effort from the AI safety community, an existential catastrophe happens, but actions taken by the AI safety community prevent such a catastrophe. In this world, what did we do to prevent the catastrophe?

Technical solutions

8 respondents considered the development of technical solutions to be important. 5 of those 8 focussed on the development of thorough safety tests for frontier models (like red-teaming, safety evaluations, and mechanistic interpretability). Such safety te...
What will the first human-level AI look like, and how might things go wrong?
by EuanMcLean — 2024-05-23T11:17:03.135Z

This is the first of 3 posts summarizing what I learned when I interviewed 17 AI safety experts about their "big picture" of the existential AI risk landscape: how will AGI play out, how things might go wrong, and what the AI safety community should be doing. See here for a list of the participants and the standardized list of questions I asked.

This post summarizes the responses I received from asking “what will the first human-level AI (HLAI) system look like?” and “what is the most likely way AI could bring about an existential catastrophe?”

Many respondents expected the first human-level AI to be in the same paradigm as current large language models (LLMs), probably scaled up, with some new tweaks and hacks, and scaffolding to make it agentic. But a different handful of people predicted that reasonably large breakthroughs are required before HLAI, and gave some interesting arguments as to why. We also talked about what those breakthroughs will be, the speed of the transition, and the range of skills such a system might have.

The most common story of how AI could cause an existential disaster was the story of unaligned AI takeover, but some explicitly pushed back on the assumptions behind the takeover story. Misuse also came up a number of times. Some took a more structural view of AI risk, emphasizing threats like instability, extreme inequality, gradual disempowerment, and a collapse of human institutions.

How to read this post

This is not a scientific analysis of a systematic survey of a representative sample of individuals, but my qualitative interpretation of responses from a loose collection of semi-structured interviews. Take everything here appropriately lightly.

Results are often reported in the form “N respondents held view X”. This does not imply that “17-N respondents disagree with view X”, since not all topics, themes and potential views were addressed in every interview. What “N respondents held view X” tells us is that at least N respondents hold X, and consider the theme of X important enough to bring up.

What will the first human-level AI look like?

Q1: What is your modal guess of what the first human-level AI (HLAI) will look like? I define human-level AI as an AI system that can carry out roughly 100% of economically valuable cognitive tasks more cheaply than a human.

There were a number of possi...
Big Picture AI Safety: Introduction
by EuanMcLean — 2024-05-23T11:15:44.037Z

tldr: I conducted 17 semi-structured interviews of AI safety experts about their big picture strategic view of the AI safety landscape: how will human-level AI play out, how things might go wrong, and what should the AI safety community be doing. While many respondents held “traditional” views (e.g. the main threat is misaligned AI takeover), there was more opposition to these standard views than I expected, and the field seems more split on many important questions than someone outside the field may infer.

What do AI safety experts believe about the big picture of AI risk? How might things go wrong, what we should do about it, and how have we done so far? Does everybody in AI safety agree on the fundamentals? Which views are consensus, which are contested and which are fringe? Maybe we could learn this from the literature (as in the MTAIR project), but many ideas and opinions are not written down anywhere, they exist only in people’s heads and in lunchtime conversations at AI labs and coworking spaces.

I set out to learn what the AI safety community believes about the strategic landscape of AI safety. I conducted 17 semi-structured interviews with a range of AI safety experts. I avoided going into any details of particular technical concepts or philosophical arguments, instead focussing on how such concepts and arguments fit into the big picture of what AI safety is trying to achieve.

This work is similar to the AI Impacts surveys, Vael Gates’ AI Risk Discussions, and Rob Bensinger’s existential risk from AI survey. This is different to those projects in that both my approach to interviews and analysis are more qualitative. Part of the hope for this project was that it can hit on harder-to-quantify concepts that are too ill-defined or intuition-based to fit in the format of previous survey work.

Questions

I asked the participants a standardized list of questions.

What will happen?

Q1 Will there be a human-level AI? What is your modal guess of what the first human-level AI (HLAI) will look like? I define HLAI as an AI system that can carry out roughly 100% of economically valuable cognitive tasks more cheaply than a human.

Q1a What’s your 60% or 90% confidence interval for the date of the first HLAI?

Q2 Could AI bring about an existential catastrophe? If so, what is the most likely way this could happen?

Q2a What’s your best guess at

...
Paper in Science: Managing extreme AI risks amid rapid progress
by JanB — 2024-05-23T08:40:40.678Z

https://www.science.org/doi/10.1126/science.adn0117

Authors:

Yoshua Bengio, Geoffrey Hinton, Andrew Yao, Dawn Song, Pieter Abbeel, Yuval Noah Harari, Ya-Qin Zhang, Lan Xue, Shai Shalev-Shwartz, Gillian Hadfield, Jeff Clune, Tegan Maharaj, Frank Hutter, Atılım Güneş Baydin, Sheila McIlraith, Qiqi Gao, Ashwin Acharya, David Krueger, Anca Dragan, Philip Torr, Stuart Russell, Daniel Kahneman, Jan Brauner*, Sören Mindermann*

Abstract:

Artificial intelligence (AI) is progressing rapidly, and companies are shifting their focus to developing generalist AI systems that can autonomously act and pursue goals. Increases in capabilities and autonomy may soon massively amplify AI’s impact, with risks that include large-scale social harms, malicious uses, and an irreversible loss of human control over autonomous AI systems. Although researchers have warned of extreme risks from AI, there is a lack of consensus about how to manage them. Society’s response, despite promising first steps, is incommensurate with the possibility of rapid, transformative progress that is expected by many experts. AI safety research is lagging. Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness and barely address autonomous systems. Drawing on lessons learned from other safety-critical technologies, we outline a comprehensive plan that combines technical research and development with proactive, adaptive governance mechanisms for a more commensurate preparation.
Power Law Policy
by Ben Turtel — 2024-05-23T05:28:46.022Z
Link: https://bturtel.substack.com/p/power-law-policy

Extreme outcomes drive tax revenue

Power Laws

Let's do a thought experiment.

The year is 1800. You’re a loan officer at a bank. Farmers come to you asking for a loan - maybe to purchase new equipment, buy more land, or make investments to improve their farm’s productivity. What are your criteria for granting a loan?

You’ll primarily want to do 2 things: avoid the losers, and reduce risk. To avoid the losers, you’ll want to learn about a farmer’s reputation and character. Are they honest and hard working? Do they have any major vices that might undermine their productivity? To reduce risk, you’ll assess the value of their collateral. How valuable is the land the bank would repossess in the case of a default?

In short, you’re much more interested in minimizing downside than you are in maximizing upside. Why? Well, giving a loan to an excellent farmer probably isn’t going to make you much more money than giving a loan to a median farmer. If a farmer has an incredible yield and produces 3x more than expected - which is very unlikely - you don’t get paid 3x the interest. Your goal is to maximize the number of loans in your portfolio that don’t default. Most of your loans are given to farmers with fairly consistent profiles - similar plans, business models, skill sets, and characteristics. You’re hoping for a consistent batting average, and you aren’t counting home runs.

Now, let’s jump to the present. Instead of a loan officer, you’re a venture capitalist, investing in early stage startups, hoping to maximize the value of your fund. How does this change your criteria?

As a VC, extreme outcomes drive the return on your fund. Accordingly, you care more about unique advantages and differentiators that give a startup a chance at becoming the winner in its category. Most investments will lose money, and many will go to zero, so the structure of financing needs allows for uncapped success when you pick a winner. You want to bet on startups that, if massively successful, could return the value of your entire fund or more.

The difference is that venture capital is ruled by power laws. Unlike a normal distribution, power laws mean that extreme outcomes drive the average, making the average very different from the median or most common outcome. This is how a VC can make money even though most investments...
Quick Thoughts on Our First Sampling Run
by jefftk — 2024-05-23T00:20:02.050Z

Cross-posted from my NAO Notebook

While the NAO has primarily
focused on wastewater, we're now spinning up a swab
sampling effort. The idea is to go to busy public places, ask
people to swab their noses, pool the swabs, sequence them, and look
for novel pathogens. We did our first collection run yesterday:

After months of planning and getting approvals, it was great to be out
there! Some thoughts on how it went:

In two hours in Kendall Square we collected 27 swabs, or
13.5/hr. I think with some optimization (below) we could do about ten
times this rate.

It was very bursty. Mostly people ignored us, I think because
they thought we were asking for money. When someone did stop to
provide a sample, and especially when multiple people stopped, they
dynamic changed dramatically. The crowd did the advertising for us,
and we had people giving us samples as fast as we could collect
them. Most of our samples came in a small number of bursts.

This means we need to be sampling at a location and time when
there's enough foot traffic that we can maintain a crowd: bringing in
each new person while the previous one is still providing a
sample. (This is very familiar from busking.)

Mid-morning in Kendall Square was not this: we were sampling at
the time that was convenient for us, which was too late for most of
the morning commuters and too early for people out getting lunch. I
suspect 8am-9am, 12-1pm, and 5pm-6pm would have been much better
times.

There are also higher foot traffic areas around Boston than
Kendall Square: we only started in Galaxy Park because that was,
again, convenient for us. In this case, however, convenience matters
enough that for our initial few iterations I think we'll stick to
Kendall.

When you don't have a crowd yet, it's really valuable for
people to be able to tell whether they want to participate from
reading the sign. I think Simon did a great job making a
professional-looking sign, but the text describing the core of what
we're asking ("swab your nose, get $5") is too small. For our next
collection run we'll print a new banner with the title and subtitle
swapped.

The banner is only readable from one side, but people don't
only come from one direction. The traditional approach is to wear a sandwich
board, but this doesn't seem very professional. I'm thinking
maybe two of these retractable banners,

...
AI Safety proposal - Influencing the superintelligence explosion
by Morgan — 2024-05-22T23:31:16.487Z

To preface, my expectation is that by default, an AI research lab will create super-intelligent AI within the next few years. Also by default, I expect it to quickly eradicate all of humanity. I would prefer if that didn't happen. I think the initiative to pause development to buy time is noble, but we still need a real solution. I do not expect that we will discover in time how to get AI models to actually care about humanity. Even if it were achievable, I would not trust AI research labs to get it right on the first shot.

I have been thinking about another method by which we might be able to survive superintelligence. This is best viewed as a rough outline of a solution. I am not at all confident it would work, but it seems to me like it could work in principle. I am very interested in feedback.

According to endosymbiotic theory, a eukaryotic cell engulfed a proto-mitochondria, but the mitochondria was useful to the cell, so the cell didn't eat it and instead formed a symbiotic relationship with it. I think we may have a chance to survive superintelligence if we follow a similar path.

The first section describes a logical progression of events that could happen during takeoff, and how we might be able to influence an AI during this window. The second section provides a sketch of an architecture for a system that could exert this influence. The third section describes how we might go about building the system in practice.

Takeoff sequence

Consider the typical hard-takeoff scenario, where an AI is just about to go into a supercritical self-improvement phase. In the most extreme hypothetical cases, this phase is very short (hours, minutes, seconds). This precludes humans from being in-the-loop. In this case, the AI's actions are dictated by the state of the world at that point in time.

This proposal attempts to influence an AI in its self-improvement phase. Intuitively, if we think of this moment as an explosion, then we can frame the problem similarly to rocketry, where we are concerned with directing that explosion. So here is a sequence of logical steps that, if they go as described, may direct in some way how the superintelligence forms.

Creating a value proposition

The first stage we need to reach is to create a software system for which an AI would get net-positive benefit from using. If using this system is a more effective way to accomplish its goals th...
Implementing Asimov's Laws of Robotics - How I imagine alignment working.
by Joshua Clancy — 2024-05-22T23:15:56.187Z

The Three Laws of Robotics

In 1942 Isaac Asimov started a series of short stories about robots. In those stories, his robots were programed to obey the three laws of robotics.

The three laws:

A robot may not injure a human being or, through inaction, allow a human being to come to harm.
A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.
A robot must protect its own existence as long as such protection does not conflict with the First or Second Law.

Humanity is finally beginning to build true humanoid robots. One may reasonably ask: What laws of robotics are our artificial intelligences obeying?

Well…

We cannot currently embed any “laws” into AI.

Sure, we can nudge our AI in certain directions via RLHF but these are not laws. These are more guidelines that can be overwritten in various unforeseen circumstances. RLHF is sticky tape, not laws written in stone.

The truth is that Modern AI does not work how Asimov or anyone at that time imagined. We do not program the AIs. Instead, we create an “evolution-like” process which “develops” the intelligence. This intelligence is then a massive complexity of weight matrices. A black box. We do not really have the control Asimov imagined. We cannot explicitly write laws into our artificial intelligences.

But… can we soon? Interpretability may hold the key.

There is a branch of research that pushes for more fine-grained understanding and control: interpretability. Interpretability promises to take that massive mess of complexity and programmatically organize it into understandable representations. We would be able to say this cluster of neurons represent “x” in “s” situation and causes “y” downstream effect.

Assuming interpretability goes well… what would it mean to embed the three laws of robotics?

In this article we are going to assume interpretability research goes well. We are then going to imagine how we could use the resulting tools to implement Asimov's laws. For now, we are going to only address the first law. In future articles, after feedback and iterations (feel free to edit at midflip.io!), we can address the other two laws.

Now lets be clear, this exercise is… crazy. This is a bit like trying to write a computer program before computers were invented. We do not know how interpretability will play out. We do not know what type of control we will have. We are def...
Higher-Order Forecasts
by ozziegooen — 2024-05-22T21:49:42.802Z

Higher-order forecasting could be a useful concept for prediction markets and forecasting systems more broadly.

The core idea is straightforward:
Nth-order forecasts are forecasts about (N-1)th order forecasts.

Abstract representation of 2nd-order forecasts. Dall-E 3

Examples

Here are some examples:

0-Order Forecasting (i.e., the ground truth)

Biden won the 2020 U.S. presidential election
The US GDP in 2023 was $27 trillion

1st-Order Forecasting (i.e., regular forecasting)

What is the chance that Trump will win the 2024 U.S. presidential election?
What will be the GDP of the US in 2024?

2nd-Order Forecasting

How much will the forecasts for US GDP in 2024 and 2025 be correlated over the next year?
How many forecasts will the question "What will be the GDP of the US in 2024?" receive in total?
If the question “What is the chance that a Republican will win the 2028 Presidential Election?” was posted to Manifold, with a subsidy of 100k Mana, what would the prediction be, after 1 month?”

3rd-Order Forecasting

How much will the forecasts, [How much will the forecasts for US GDP in 2024 and 2025 be correlated over the next year?] and [How many forecasts will the question "What will be the GDP of the US in 2024?" receive in total?], be correlated, from now until 2024?
How valuable were all the forecasts for the question, [‘How many forecasts will the question "What will be the GDP of the US in 2024?" receive in total?’]

As forecasting systems mature, higher-order forecasts could play a role analogous to financial derivatives in markets. Derivatives allow for more efficient pricing, risk transfer, and information aggregation by letting market participants express views on the relationships between assets. Similarly, higher-order forecasts could allow forecasters to express views on the relationships between predictions, leading to a more efficient and informative overall forecasting ecosystem.

Benefits

Some potential benefits of higher-order forecasting include:

Identify Overconfidence

Improve the accuracy of forecasts by having participants directly predict and get rewarded for estimating overconfidence or poor calibration in other forecasts.
"How overconfident is [forecast/forecaster] X"

Prioritize Questions

Prioritize the most important and decision-relevant questions by forecasting the value of information from different predictions.
"How valuable is the information from forecasting

...
A Positive Double Standard—Self-Help Principles Work For Individuals Not Populations
by James Stephen Brown — 2024-05-22T21:37:16.578Z

This story is reposted from

nonzerosum.games

where it appears in it’s intended form, full colour with functioning interactive elements, jump over to the site for the authentic experience.

I have a confession: As a teenager I was desperate to learn how to win friends and influence people, to think and grow rich and develop the habits of highly effective people (seven to be precise)-I am James and I am a self-help-oholic.

These tomes held for me the promise of fame and fortune (though they would earnestly profess otherwise) but most importantly they offered a salve for my crippling shyness.

PERSONAL RESPONSIBILITY

While I am, today, 20 years self-help-sober, the genre is as popular as ever-with its imperative to stand up straight with your shoulders back (Jordan Peterson), to take extreme ownership (Jocko Willink) and exercise, subtly, the art of not giving a f*ck (Mark Manson).

Though not all the messages are as hyper-masculine as these, they all encourage the seemingly uncontroversial virtue of ‘personal responsibility’. I will argue that, regarding ‘personal responsibility’, we should employ a positive double standard.

WHAT IS A POSITIVE DOUBLE STANDARD?

It is easy to hold a double standard where we expect others to take personal responsibility, while excusing our own shortfalls.

A positive double standard requires the opposite-that I take personal responsibility while not expecting the same of society. This position holds that we have the power to change our own individual futures, but recognises the statistical realities of social dynamics on populations.

FACTS

The power of ‘personal responsibility’ to positively impact one’s own life has some scientific support. Studies on

locus of control

, developed by Julian B. Rotter in 1954, found that subjects who believed they could influence life’s outcomes through action tended to attain better job satisfaction, more successful careers, and better stress management.

Albert Bandura’s

research in the 70s and 80s

found that individuals who believed in their ability to influence events affecting their lives were more likely to set challenging goals and persist longer in the face of obstacles and achieve success. Personal responsibility also underpins much of the well-established field of Cognitive Behavioural Therapy as well as Carol Dweck’s

growth mindset

.

A PHILOSOPHICAL PERSPECTIVE

The imperative to take personal responsibilit...