trentmkelly/LessWrong-43k huggingface dataset:
The Polarity Problem [Draft]
by Dan H — 2023-05-23T21:05:34.567Z

This is a draft written by Cameron Domenico Kirk-Giannini, assistant professor at Rutgers University, and Simon Goldstein, associate professor at the Dianoia Institute of Philosophy at ACU, as part of a series of papers for the Center for AI Safety Philosophy Fellowship's midpoint. Dan helped post to the Alignment Forum. This draft is meant to solicit feedback. The authors would especially welcome pointers to work related to the topic which they have not cited.

Abstract:

If it is possible to construct artificial superintelligences, it is likely that they will be extremely powerful. A natural question is how many such superintelligences to expect. Will the future be shaped by a single superintelligence (a unipolar outcome), or will there be multiple superintelligences shaping the future through their cooperative or adversarial interactions (a multipolar outcome)? We refer to this question as the polarity problem. This paper investigates the polarity problem. First, we consider the question of safety, suggesting that multipolar outcomes are likely to be safer for humanity than unipolar outcomes because humans are less likely to be disempowered in multipolar scenarios. Then we develop a series of causal models of AI agents that make predictions about the relative likelihood of unipolar as opposed to multipolar outcomes. Central to our models are three parameters: the time it takes for an AI attacker to develop an attack against an AI defender, the time it takes for the AI defender to develop its defense, and how far the defender lags behind the attacker temporally. We use our models to identify possible interventions which could change the probability of a multipolar outcome, thereby leading to a safer trajectory for AI development.

1. The Question

If it is possible to construct artificial superintelligences, it is likely that they will be extremely capable actors. Indeed, artificial superintelligences will likely be so much more capable than traditional actors that if they come into existence and function autonomously to gain and maintain power, the future will be shaped more or less exclusively by them.

If the future may be shaped more or less exclusively by artificial superintelligences, a natural question is how many such superintelligences to expect. Will the future be shaped by a single extremely powerful artificial superintelligence, or will there ...
Progress links and tweets, 2023-05-23
by jasoncrawford — 2023-05-23T20:15:49.360Z
Link: https://rootsofprogress.org/links-and-tweets-2023-05-23

The Progress Forum

Why growth in developing countries matters for progress (via @jmazda)

Opportunities

Dwarkesh Patel is looking for a physics tutor

Announcements

Arnold Ventures is launching a major new infrastructure initiative
“Building a Better NIH,” a paper series from IFP and others (via @calebwatney)
New book from Daron Acemoglu & Simon Johnson against “techno-optimism.” I expect to learn a lot from this, and to find much to agree and to disagree with
Blueprint for a new Great Exhibition (by @antonhowes)

News

RIP Robert Lucas. His most-quoted line: “The consequences for human welfare involved in questions [of economic growth] are simply staggering: Once one starts to think about them, it is hard to think about anything else.” More from @srajagopalan, @lugaricano, @singhabhi, and many others.
NASA: Blue Origin will build a second Human Landing System for the Moon
Longevity startup NewLimit raises $40M (via @garrytan)

Articles

How will AI impact science? (by @michael_nielsen)
Britain shows what degrowth looks like in practice (via @s8mb). So does South Africa
King Tut and his meteorite dagger (by @WillRinehart)

Queries

A case against AI x-risk by someone who understands the case for x-risk well?
Are there any good examples of “nuclearpunk”?
Technologies that were expected to have huge impacts but never got cheap?
Which public figure combines intellectual rigor and moral forcefulness?
Who decides what kinds of business are good for private equity?

Tweets

What if they gave an Industrial Revolution and nobody came? (the intro to my recent post, ICYMI). Also: The American Information Revolution in Global Perspective
How people without research training can push forward the frontier of technology
Everything has to be invented. Even spaces and punctuation
How to fight antibiotic resistance. Also, human DNA can be sequenced from the air
Nuclear fracking (!) But, our nation’s nuclear reactor laboratory has gone 56 years without building a new reactor
Landmark Experiments in Twentieth-Century Physics
Self-imposed challenges (sports, games, music) are sufficient for human flourishing
Regulatory costs explain ~⅓ of the increase in market power in the last 50 years. Also, whatever AI regulations we write today will sound ridiculous 50 years from now
Learning exactly how San Francisco governance works. Related, most people pay no attention to the deficit or how it constrains political choices
T

...
Yoshua Bengio: How Rogue AIs may Arise
by harfe — 2023-05-23T18:28:27.489Z
Link: https://yoshuabengio.org/2023/05/22/how-rogue-ais-may-arise/

[I think this is noteworthy, as Yoshua Bengio is a famous ML expert, and he presents his arguments for AI xrisk here. I copied the whole blogpost below.]
The rise of powerful AI dialogue systems in recent months has precipitated debates about AI risks of all kinds, which hopefully will yield an acceleration of governance and regulatory frameworks. Although there is a general consensus around the need to regulate AI to protect the public from harm due to discrimination and biases as well as disinformation, there are profound disagreements among AI scientists regarding the potential for dangerous loss of control of powerful AI systems, also known as existential risk from AI, that may arise when an AI system can autonomously act in the world (without humans in the loop to check that these actions are acceptable) in ways that could potentially be catastrophically harmful. Some view these risks as a distraction for the more concrete risks and harms that are already occurring or are on the horizon. Indeed, there is a lot of uncertainty and lack of clarity as to how such catastrophes could happen. In this blog post we start a set of formal definitions, hypotheses and resulting claims about AI systems which could harm humanity and then discuss the possible conditions under which such catastrophes could arise, with an eye towards helping us imagine more concretely what could happen and the global policies that might be aimed at minimizing such risks.

Definition 1: A potentially rogue AI is an autonomous AI system that could behave in ways that would be catastrophically harmful to a large fraction of humans, potentially endangering our societies and even our species or the biosphere.

Executive Summary

Although highly dangerous AI systems from which we would lose control do not currently exist, recent advances in the capabilities of generative AI such as large language models (LLMs) have raised concerns: human brains are biological machines and we have made great progress in understanding and demonstrating principles that can give rise to several aspects of human intelligence, such as learning intuitive knowledge from examples and manipulating language skillfully. Although I also believe that we could design AI systems that are useful and safe, specific guidelines would have to be respected, for example limiting their agency. On the other hand the recent advances sug...
'Fundamental' vs 'applied' mechanistic interpretability research
by Lee Sharkey — 2023-05-23T18:26:18.174Z

When justifying my mechanistic interpretability research interests to others, I've occasionally found it useful to borrow a distinction from physics and distinguish between 'fundamental' versus 'applied' interpretability research.

Fundamental interpretability research is the kind that investigates better ways to think about the structure of the function learned by neural networks. It lets us make new categories of hypotheses about neural networks. In the ideal case, it suggests novel interpretability methods based on new insights, but is not the methods themselves.

Examples include:

A Mathematical Framework for Transformer Circuits (Elhage et al., 2021)
Toy Models of Superposition (Elhage et al., 2022)
Polysemanticity and Capacity in Neural Networks (Scherlis et al., 2022)
Interpreting Neural Networks through the Polytope Lens (Black et al., 2022)
Causal Abstraction for Faithful Model Interpretation (Geiger et al., 2023)
Research agenda: Formalizing abstractions of computations (Jenner, 2023)
Work that looks for ways to identify modules in neural networks (see LessWrong 'Modularity' tag).

Applied interpretability research is the kind that uses existing methods to find the representations or circuits that particular neural networks have learned. It generally involves finding facts or testing hypotheses about a given network (or set of networks) based on assumptions provided by theory.

Examples include

Steering GPT-2-XL by adding an activation vector (Turner et al., 2023)
Discovering Latent Knowledge in Language Models (Burns et al., 2022)
The Singular Value Decompositions of Transformer Weight Matrices are Highly Interpretable (Millidge et al., 2022)
In-context Learning and Induction Heads (Olsson et al., 2022)
We Found An Neuron in GPT-2 (Miller et al., 2023)
Language models can explain neurons in language models (Bills et al., 2023)
Acquisition of Chess Knowledge in AlphaZero (McGrath et al., 2021)

Although I've found the distinction between fundamental and applied interpretability useful, it's not always clear cut:

Sometimes articles are part fundamental, part applied (e.g. arguably 'A Mathematical Framework for Transformer Circuits' is mostly theoretical, but also studies particular language models using new theory).
Sometimes articles take generally accepted 'fundamental' -- but underutilized -- assumptions and develop methods based on them (e.g. Causal Scrubbing, where the

...
Coercion is an adaptation to scarcity; trust is an adaptation to abundance
by Richard_Ngo — 2023-05-23T18:14:19.117Z

There’s a sense in which I’m postulating a trillion dollar bill lying on the ground. If the skills I’ve talked about so far in this sequence would lead to so much more flourishing, why haven’t they become far more common? I’ve already given a partial answer: that our evolutionary environment was much more dangerous than our current environment. But I want to extend this to a more general answer: that coercion is an adaptation to scarcity; and that we only very recently left the era in which scarcity was the dominant feature of people’s lives.

Under conditions of scarcity, you don’t have enough slack that you can afford to take risks. If misbehavior from any individual in the group would risk the lives of many others, you have to coerce them into staying in line; if a loss on any gamble would leave you ruined, then you need to avoid taking those gambles even when they’re winning in expectation. (Poker players think a lot about this when trying to manage their bankroll—if a game has high-enough stakes that they’d be out of money if they lost, they have to avoid it, even if they expect to make money there on average.)

By contrast, in an abundant environment, you can take the optimal long-term strategy, even if there’s a risk that it’ll leave you way down in the short term. In particular, you can put effort into building trust with others, even though that leaves you more vulnerable to being let down or betrayed. With that trust, you can receive a huge range of gains from cooperation.[1]

Western societies are incredibly abundant in many ways. As a citizen, you face almost zero risk of starvation, dying in a war, or exile from your country; meanwhile deaths from most diseases and accidents are dramatically lower than in the past. There’s more career flexibility than there ever has been before: there are many routes to success, including self-employment. And society is far richer than it’s ever been before: the median person in a western society is incredibly wealthy both by historical standards and by the standards of most people across the world. Even welfare recipients are still well off by historical and global standards.

What does it look like to internalize a feeling of abundance? It might involve not forcing yourself to study when you don’t feel like it; or being less defensive when people criticize you; or deciding to be less harsh on yourself about the ...
Is "brittle alignment" good enough?
by the8thbit — 2023-05-23T17:35:43.750Z

I'm new here, so I apologize if this is a common question, or if I'm using the wrong terms, or the wrong framework for approaching this. I also apologize if this post is badly structured. When I talk about alignment, it feels like so many of the ideas have rapidly branching dependencies, and sometimes fold back in on themselves, so it makes it hard for me to talk about in a coherent way.

Whenever I see people discuss bad alignment, it seems to be regarding agents that either immediately perform an attack, or, in the slightly more sophisticated thought experiment, are secretly antagonistic, but outwardly aligned until a future moment where they know they can successfully pursue their true reward function.

Instead, I would like to investigate the possibility of an agent which both appears aligned, and internally "believes" itself to be aligned (it is not deceptive), and yet, it becomes unaligned at some later point.

With a hashing algorithm, when you hash a sequence, in principle, you receive a unique value for any given input. However, in practice, all hashing algorithms have a very small chance of a collision, where one input produces the same output hash as some completely unrelated input.

Is it possible for an agentic system's reward function to have a similar vulnerability? If we assume an architecture composed of perceptrons and trained weights, through which we take some input, route it through the weighted synapses, and produce some output, would it be surprising if its possible to find some arbitrary action which produces the same or greater reward than the action we targeted in training?

Imagine an AI optimized to produce as many paperclips as possible, and then one day it begins producing rubber ducks instead. Not because our training actually targeted a deceptive agent, (let's say we develop some powerful interpretability tools that allow us to see if an agent is deceptive during training), or because this is an emergent intermediate goal, but because our completely honest and aligned agent suddenly discovered that creating rubber ducks produced greater reward for the same effort, and became unaligned at that moment.

Maybe it discovers this collision because another system, perhaps even a less general/powerful system, trained as an adversary to the first, delivers the information to it. Or maybe the first system discovers this itself, or through some sub...
Will Artificial Superintelligence Kill Us?
by James_Miller — 2023-05-23T16:27:52.419Z

For Open Philanthropy AI Worldviews Contest

Introduction

Through advancements in machine learning, we are progressing towards the development of artificial superintelligence (ASI). As machine learning often yields opaque results, our predictions about ASI behavior in novel situations are uncertain. Alarmingly, some theories suggest that ASI might eliminate humans to secure resources for its own use.

Even if ASI doesn't intentionally harm us, our existence could be endangered if our fundamental necessities, like clean air, water, and a stable climate, interfere with the ASI's optimal functioning. Additionally, intense conflicts among ASIs could render the Earth uninhabitable for humans.

Market forces drive companies to chase ASI development, even if they believe that ASI could cause human extinction. This mainly occurs as every company understands that halting ASI research could give their competitors an edge. Stopping the global pursuit of ever more powerful AI seems unlikely, given the growing financial and military advantages attached to it. In addition, older political leaders, who might benefit from the potential life-extending effects of ASI, could push for rapid ASI development, despite the substantial risk of human extinction.

I believe our best chance at survival depends on the possibility that even unaligned ASIs might see a practical benefit of preserving humanity and sharing a small part of the universe's resources with us. Predicting our survival chances is challenging, but I cautiously estimate them to be between ten and forty percent.

This paper explores the important issues related to ASI risk, evaluates potential differences in expert opinions on each subject, and points out areas for future research to improve our comprehension of this complicated scenario.

The Orthogonality Thesis

The orthogonality thesis suggests there is no link between an ASI's intelligence and its goals, suggesting that advanced ASIs could strive to fulfill any objective, even something as trivial as maximizing the number of paperclips in the universe.[1] As such, an ASI might annihilate humanity for a marginal enhancement in achieving its goal. Conversely, if the thesis is flawed, we can hope that with increasing intelligence, agents would naturally align with moral values that respect sentient life, thus avoiding human annihilation.

Renowned computer scientis...
Phone Number Jingle
by jefftk — 2023-05-23T15:20:01.874Z

When the kids were little and we were at the kind of event where they
might get separated from us, I'd write my phone number on tape
and stick it to their backs:

Anna at NEFFA 2019, age 3

The kids are getting older now, and are playing independently enough
that it would be good if they always had my phone number. Lily (9yo)
has had a piece of paper in her backpack for a while now, but unless
she's going to or from school she won't have it with her. Several
rounds of "repeat after me" weren't working, so this weekend we made
up a song for it. After just a few times singing it she has it down.

This may be a trick that works unusually well for Lily, since she's
pretty musical, but the popularity of jingles
for phone numbers in advertising makes me think this would work
for others?

Comment via: facebook, mastodon
GPT4 is capable of writing decent long-form science fiction (with the right prompts)
by RomanS — 2023-05-23T13:41:10.105Z

I like to experiment with AI-assisted fiction writing. Today, for the first time, I've noticed that GPT4 is actually getting good at the task.

Perhaps it's a result of some updates by OpenAI, or maybe I got the prompts right, but GPT4 finally wrote a science fiction novella worth reading.

As the initial prompt, I used a lengthy contraption described in another post, with the following plot summary:

a group of scientists has discovered that Troodon dinosaurs were intelligent species who have created a technologically advanced civilization, suddenly destroyed. The year-long path to the scientific discovery starts with the group stumbling upon a strange outlier in their geological measurements.

For the rest of the story, I just repeatedly asked GPT4 to write the next chapter (sometimes providing the chapter's title to steer the plot in the direction I want). And in a few instances I asked it to rewrite a twist I didn't like, or to put more emphasis on something.

Below is the unedited result (with the prompts marked).

The whole thing is especially amazing given that I'm a non-native English speaker of no writing skills. And it took only a couple of hours to complete my (our? its?) first full-sized novella.

Let's call it "The Troodon Quill".

Chapter 1: Echoes in the Stone

The low thrum of the spectrometer filled the narrow confines of the cave, a harmonious hum that sung its serenade to the rocks. Dr. Ada Worthington, dressed in the sombre uniform of a seasoned palaeontologist, her dirty-blonde hair escaping from beneath her hat, listened to the resonating symphony, eyes closed. An outsider might mistake it for a moment of relaxation. They would be wrong. Ada was anything but relaxed.

“We're looking for anomalies, Ada,” Dr. Leo Hartman had said at the outset of this mission. Leo was her mentor, a titan in the field of palaeontology, whose reputation carried the weight of numerous groundbreaking discoveries and published works. He was a seeker of truths locked away in layers of rock, dust and time.

Ada remembered his words. She was looking for anomalies in the symphony. A slight deviation in the humming, the hint of an offbeat note.

Patience is a virtue in our line of work, Leo had said, his words spoken with a reverent love for the stories the earth preserved beneath its surface.

As Ada listened to the music of the spectrometer, she thought of he...
Do humans still provide value in correspondence chess?
by Jonathan Paulson — 2023-05-23T12:15:13.462Z

In chess, AIs are very superhuman; the best players in the world would lose nearly every game against any modern computer player.

Do humans still have something to add? The continued existence of correspondence chess, IMO, suggests that they do. In correspondence chess players have days to make each move, and play from their homes. Due to the impossibility of policing cheating under these conditions, correspondence players are allowed to use computer assistance.

You might think this would make the games just a question of who has more computing power. But as far as I can tell, that’s not the case.

What are humans adding? Low confidence, but I think it’s mostly opening prep; try to find a line that looks ok on shallow computer analysis, but where deeper analysis shows you have an advantage. The human value-add is telling the computer which lines to analyze. Since the chess game tree is so large, advice like this is quite valuable.

Proof that correspondence chess is still played: https://www.nytimes.com/2022/11/09/crosswords/correspondence-chess.html
Interview with a human player (from 2016): https://en.chessbase.com/post/better-than-an-engine-leonardo-ljubicic-1-2

On the other hand: I don’t play correspondence chess, so I’m not that confident in the claims above. And some people don’t find them plausible: https://twitter.com/liron/status/1660890927920201728?s=46&t=UlLg1ou4o7odVYEppVUWoQ

Why should we care? This might provide some indication of what value humans can provide in a world of superhuman AI (at least initially).

Can anyone provide a more definitive account of what value, if any, humans add in correspondence chess?