trentmkelly/LessWrong-43k huggingface dataset:
[Linkpost] Personal and Psychological Dimensions of AI Researchers Confronting AI Catastrophic Risks
by Bogdan Ionut Cirstea — 2023-08-12T22:02:09.895Z

This is a linkpost for https://yoshuabengio.org/2023/08/12/personal-and-psychological-dimensions-of-ai-researchers-confronting-ai-catastrophic-risks.

Yoshua Bengio:

For most of these years, I did not think about the dual-use nature of science because our research results seemed so far from human capabilities and the work was only academic. It was a pure pursuit of knowledge, beautiful, but mostly detached from society until about a decade ago. I now believe that I was wrong and short-sighted to ignore that dual-use nature. I also think I was not paying enough attention to the possibility of losing control to superhuman AIs.

[...] it started to dawn on me that my previous estimates of when human-level AI would be reached needed to be radically changed. Instead of decades to centuries, I now see it as 5 to 20 years with 90% confidence.
And what if it was, indeed, just a few years?

I started reading more about AI safety and came to a critically important conclusion: we do not yet know how to make an AI agent controllable and thus guarantee the safety of humanity! And yet we are – myself included until now – racing ahead towards building such systems.

It is painful to face the idea that we may have been contributing to something that could be greatly destructive. Human nature will lead us towards brushing aside these thoughts or finding comfort in reassuring arguments rather than face the full horror of such possibilities. Bringing the benefits of AI to the table is not sufficient to compensate if the possible negative outcomes include catastrophic misuses of AI on par with nuclear war and pandemics, or even existential risk.

As scientists, we should avoid making claims we can’t support; but as decision-makers we also ought to act under uncertainty to take precautions. In spite of our differences in points of view, it’s time for our field of AI to seriously discuss the questions: what if we succeed? What if potentially dangerous superhuman AI capabilities are developed sooner than expected? Let’s embrace these challenges and our differences, while being mindful of each other’s humanity and our unique emotional and psychological journeys in this new era of AI.
The Benevolent Ruler’s Handbook (Part 2): Morality Rules
by FCCC — 2023-08-12T14:25:58.642Z

Read Part 1 for more background.

Let’s get on track

If we’re going to understand what constitutes good policy, we need to understand what constitutes a good moral theory. We can gain some insights from looking at the practices of moral philosophy. Yes, you knew damn well that when we started this, we would talk about the trolley problem.

So you’re in class, and your philosophy professor asks you why you chose to pull the lever, you say “More lives are better than fewer”, and then they hit you with the Surgeon variant of the problem. Why do they do that? What are we achieving when we pose moral dilemmas?

Pictured: My partner explaining why reality TV is enjoyable The Good Place’s take on the trolley problem

Your professor is, for each dilemma, getting you to state your intuitive moral preferences over two specified timelines (one in which you pull the lever, and the one in which you don’t). Then you’re supposed to state a rule that consistently represents your moral preferences over all possible dilemmas. And if your proposed rule fails to account for the next contrived dilemma your professor throws at you, you’ve done something wrong. It’s essentially the same as scientific falsification, except we use thought experiments.

People in the philosophy biz often get away with questions that no one else can ask

Now you might wonder, if moral intuitions arbitrate which moral rules are “correct”, why not just use moral intuition to evaluate these dilemmas? What is the point of a moral rule? Well, they have several purposes:

Unless we’re born with moral intuitions that are perfectly accurate, we’re going to have some priors that are wrong.

Sometimes our moral priors are outright contradictory. For example, “Violence is never moral” and “Culture determines morality” contradict because of the possibility of violent cultures.

Sometimes our stated justifications for our moral intuitions are contradictory, but those intuitions could be represented with a non-contradictory moral rule.

Often, our intuitive ranking will be incomplete, which makes moral decisions difficult even if we could perfectly predict the consequences.

Finding a moral rule helps us to correct poor moral intuitions, explain our good moral intuitions with more clarity, and guide us where our intuitions don’t exist.

Our view of thinking of moral rules as summarising moral preferences over timelines works not...
Learning as you play: anthropic shadow in deadly games
by dr_s — 2023-08-12T07:34:42.261Z

Epistemic status: I'm not a mathematician, but this is simple stuff and I think I double checked all my calculations well enough. Criticism is welcome!

Some time ago a post came out claiming that "anthropic shadow is reflectively inconsistent". I want to argue the exact opposite thesis: anthropic shadow is a trivially true phenomenon, and it is in fact just a special case of a phenomenon that applies to a wide class of possible games, and is fundamentally unavoidable.

To put forward a formal thesis:

Given a game G that possesses the following characteristics:

there is some aspect of the rules that is unknown to the player, and can not be deduced by observing other games (either because there are no other games, or because the rules are fully randomized every time);
said unknown aspect of the rules affects in some way the player's ability to gather information about the game by playing, e.g. by abruptly ending it, or by obfuscating outcomes;

then the average player will not be able to play G optimally based only on in-game observations.

It is pretty trivial stuff if one thinks about it; obviously, if the unknown rules mess with our ability to learn them, that reflective quality fundamentally rigs the game. The most obvious example of this are what I'll call "deadly games" - namely, games in which one outcome abruptly ends the game and thus makes all your accumulated experience useless. This property of such games however is fundamental to their self-reflexive nature, and it has nothing to do with death per se. If instead of dying you are kicked out of the game, the result is the same. "Anthropic undeath" doesn't fix this: the fundamental problem is that you start the game ignorant of some crucial information that you need to play the game optimally, and you can't learn that information by playing because any event that would give you valuable bits to update your beliefs reflexively messes with your ability to observe it. Therefore, your sampling of a game trajectory from inside a game will always be incomplete, and if information can't be carried from one game to another, this will be a chronic, unfixable problem. You can't play at your best because the deck is stacked, you don't know how, and the dealer reshuffles new cards in every time.

A game of Chinese Roulette

Let's come up with a game that we'll call Chinese Roulette. This game is a le...
Biological Anchors: The Trick that Might or Might Not Work
by Scott Alexander — 2023-08-12T00:53:30.159Z
Link: https://astralcodexten.substack.com/p/biological-anchors-a-trick-that-might

This post originally posted on Astral Codex Ten on Feb 23 2022.

It was printed in The Carving of Reality, the third volume of the Best of LessWrong book series. It was included as a (shorter) replacement for Ajeya Cotra's Draft report on AI timelines, and Eliezer's Biology-Inspired AGI Timelines: The Trick That Never Works, covering the topic from multiple sides.

It's crossposted here with Scott's permission for completeness (i.e. having all essays in the book appear on LessWrong).

Introduction

I've been trying to review and summarize Eliezer Yudkowksy's recent dialogues on AI safety. Previously in sequence: Yudkowsky Contra Ngo On Agents. Now we’re up to Yudkowsky contra Cotra on biological anchors, but before we get there we need to figure out what Cotra's talking about and what's going on.

The Open Philanthropy Project ("Open Phil") is a big effective altruist foundation interested in funding AI safety. It's got $20 billion, probably the majority of money in the field, so its decisions matter a lot and it’s very invested in getting things right. In 2020, it asked senior researcher Ajeya Cotra to produce a report on when human-level AI would arrive. It says the resulting document is "informal" - but it’s 169 pages long and likely to affect millions of dollars in funding, which some might describe as making it kind of formal. The report finds a 10% chance of “transformative AI” by 2031, a 50% chance by 2052, and an almost 80% chance by 2100.

Eliezer rejects their methodology and expects AI earlier (he doesn’t offer many numbers, but here he gives Bryan Caplan 50-50 odds on 2030, albeit not totally seriously). He made the case in his own very long essay, Biology-Inspired AGI Timelines: The Trick That Never Works, sparking a bunch of arguments and counterarguments and even more long essays.

There's a small cottage industry of summarizing the report already, eg OpenPhil CEO Holden Karnofsky's article and Alignment Newsletter editor Rohin Shah's comment. I've drawn from both for my much-inferior attempt.

Part I: The Cotra Report

Ajeya Cotra is a senior research analyst at OpenPhil. She's assisted by her fiancee Paul Christiano (compsci PhD, OpenAI veteran, runs an AI alignment nonprofit) and to a lesser degree by other leading lights. Although not everyone involved has formal ML training, if you care a lot about whether efforts are “establishment” or “contrarian”,...
Simulate the CEO
by robotelvis — 2023-08-12T00:09:44.105Z
Link: https://messyprogress.substack.com/p/simulate-the-ceo

Humans can organize themselves into remarkably large groups. Google has over a hundred thousand employees, the worldwide Scouting movement has over fifty million scouts, and the Catholic Church has over a billion believers.

So how do large numbers of people coordinate to work towards a common mission?

Most organizations are headed by some kind of “CEO” figure. They may use a title like President, Executive Director, or Pope, and their power is likely constrained by some kind of board or parliament, but the basic idea is the same - there is a single person who directs the behavior of everyone else.

If you have a small group of people then the CEO can just tell each individual person what to do, but that doesn’t scale to large organizations. Thus most organizations have layers of middle managers (vicars, moderators, regional coordinators, etc) between the CEO and regular members.

In this post I want to argue that one important thing those middle managers do is that they “simulate the CEO”. If someone wants to know what they should be doing, the middle manager can respond with an approximation of the answer the CEO would have given, and thus allow a large number of people to act as if the CEO was telling them what to do.

This isn’t a perfect explanation of how a large organization works. In particular, it ignores the messy human politics and game playing that makes up an important part of how companies work and what middle managers do. But I think CEO-simulation is a large enough part of what middle managers do that it’s worth taking a blog post to explore the concept further.

In particular, if simulating the CEO is a large part of what middle managers do, then it’s interesting to think about what could happen if large language models like GPT get good at simulating the CEO

A common role in tech companies is the Product Manager (PM). The job of the a PM

Is to cause the company to do something (eg launch a product) that requires work from multiple teams. Crucially, the PM does not manage any of these teams and has no power to tell any of them what to do.

So why does anyone do what the PM asks?

Mostly, it’s because people trust that the PM is an accurate simulation of the CEO. If the PM says something is important, then the CEO thinks it is important. If the PM says the team should do things a particular way then the CEO would want them to do it that way....
How to decide under low-stakes uncertainty
by dkl9 — 2023-08-11T18:07:50.133Z
Link: https://dkl9.net/essays/mental_coinflip.html

Say you're stuck in uncertainty between two actions you're considering. They seem about equally good, but you suspect one is better, and it's not obvious which. You already have all the information to obviously collect about the problem.

For situations where getting it right really matters, try harder to get more information, and use methods more reliable than those presented here.

For lower-stakes problems:

Assign one option to 0/tails and the other to 1/heads.
Flip a coin, i.e. use a one-bit random number generator.
Start accepting the decision from the coinflip, and observe your revealed unconscious intuition, for you will probably have some.
If that intuition opposes the coin's decision, go with the intuition. Otherwise, you either actively came to agree with the random decision, or it still seems terribly uncertain, so you go with the random choice.

This is classic advice. I'm just sharing it. The "innovation" here is a way to handle cases where it's inconvenient to flip a coin or do the equivalent.

Mental coinflip

Assign one option to 0/tails and the other to 1/heads.
Think about something unrelated.
Notice the first word that pops into your mind.
Count letters in that word. If it's even, you get 0; if it's odd, you get 1.
Use that 0 or 1 to continue at step 3 of the general procedure.

This is very crude randomness, but it doesn't really have to be random, just uncorrelated with the topic of decision. You should not use a mental coinflip to try to run a fair game of chance.

There may be a bias in the length-parity of words you think of. I, so far, haven't noticed one. But that wouldn't necessarily be a problem. Once you notice such a bias, you can exploit it by assigning actions to 0/1 so that the mental coinflip bias opposes a bias you naturally have.

E.g. if you pick odd-length words more often, when the choice is between "doing nothing" and "doing something", assign "doing something" to 1. Laziness may lead you to want to "do nothing" more often than you should.

More than two options

You can use the same methods here for decisions more complicated than a dichotomy.

List the options you're considering.
Split that list into two subsets.
Apply the coinflip-assisted decision procedure to pick one subset.
If the selected subset has just one item, that's your decision. Otherwise, repeat this procedure on the selected subset.
The Pandemic is Only Beginning: The Long COVID Disaster
by salvatore mattera — 2023-08-11T17:36:10.656Z

I wrote this for my Substack. Posting it here as I hope that I am wrong, and I'm looking for people to debunk my arguments. I found this post and this post, but both are old and contain numerous factual errors (I don't think they were wrong at the time of writing, but better data has come out since they were written. The point of this post isn't to debunk them directly, and I can obviously expand if requested, but for example, we now know that long COVID in children appears quite common).

Imagine a world where hundreds of millions of people are too sick to work. Could society continue to function?

It may sound hyperbolic, but barring some incredible fortune

1

, it looks like that’s where we are headed: A decade from now, almost one third of the world’s population could suffer from a severe chronic illness — the consequence of repeated COVID infections.

This condition, known as long COVID or post-COVID, isn’t rare. It affects many people who catch COVID. Even if they caught COVID before. Even if they are young and healthy. Even if they are vaccinated (or not vaccinated).

There are still a lot of unknowns, but the story that’s emerging goes like this:

Given that COVID is highly contagious, and immunity short-lasting, every person on the planet will probably catch COVID many times over their course of their life.
Every time a person catches COVID, there’s a chance they will develop long COVID. So with each reinfection, the cumulative probability of developing long COVID increases.
All people with long COVID suffer from a reduction in their quality of life, but some are worse than others. Many people with long COVID are too disabled to work.
Some people recover from long COVID, but for others it may be lifelong. And even when people do recover, a COVID reinfection often triggers a relapse back into long COVID. Reinfections can also make existing long COVID more severe.

Over time, as people rack up more and more COVID reinfections, the total number of people suffering from long COVID will increase. Some of these people will be too sick to hold a job. More and more people will be forced to turn to government disability programs. The labor force will shrink. Healthcare systems, and society in general, will be tremendously strained.

What is long COVID, and what causes it?

There are famous actors (

source

), athletes (

source

), politicians (

source

), and entertainers (

source

) who hav...
When discussing AI risks, talk about capabilities, not intelligence
by Vika — 2023-08-11T13:38:48.844Z
Link: https://vkrakovna.wordpress.com/2023/08/09/when-discussing-ai-risks-talk-about-capabilities-not-intelligence/

Public discussions about catastrophic risks from general AI systems are often derailed by using the word “intelligence”. People often have different definitions of intelligence, or associate it with concepts like consciousness that are not relevant to AI risks, or dismiss the risks because intelligence is not well-defined. I would advocate for using the term “capabilities” or “competence” instead of “intelligence” when discussing catastrophic risks from AI, because this is what the concerns are really about. For example, instead of “superintelligence” we can refer to “super-competence” or “superhuman capabilities”.

Image source: TED talks

When we talk about general AI systems posing catastrophic risks, the concern is about losing control of highly capable AI systems. Definitions of general AI that are commonly used by people working to address these risks are about general capabilities of the AI systems:

PASTA definition: “AI systems that can essentially automate all of the human activities needed to speed up scientific and technological advancement”.
Legg-Hutter definition: “An agent’s ability to achieve goals in a wide range of environments”.

We expect that AI systems that satisfy these definitions would have general capabilities including long-term planning, modeling the world, scientific research, manipulation, deception, etc. While these capabilities can be attained separately, we expect that their development is correlated, e.g. all of them likely increase with scale.

There are various issues with the word “intelligence” that make it less suitable than “capabilities” for discussing risks from general AI systems:

Anthropomorphism: people often specifically associate “intelligence” with being human, being conscious, being alive, or having human-like emotions (none of which are relevant to or a prerequisite for risks posed by general AI systems).
Associations with harmful beliefs and ideologies.
Moving goalposts: impressive achievements in AI are often dismissed as not indicating “true intelligence” or “real understanding” (e.g. see the “stochastic parrots” argument). Catastrophic risk concerns are based on what the AI system can do, not whether it has “real understanding” of language or the world.
Stronger associations with less risky capabilities: people are more likely to associate “intelligence” with being really good at math than

...
What are the flaws in this AGI argument?
by William the Kiwi  — 2023-08-11T11:31:12.690Z

Aligned AGI is a large scale engineering task

Humans have never completed at large scale engineering task without at least one mistake

An AGI that has at least one mistake in its alignment model will be unaligned

Given enough time, an unaligned AGI will perform an action that will negatively impact human survival

Humans wish to survive

Therefore, humans ought not to make an AGI until one of the above premises changes.

This is another concise argument around AI x-risk. It is not perfect. What flaw in this argument do you consider the most important?
Google DeepMind's RT-2
by SandXbox — 2023-08-11T11:26:16.599Z
Link: https://robotics-transformer2.github.io/

Abstract

We study how vision-language models trained on Internet-scale data can be incorporated directly into end-to-end robotic control to boost generalization and enable emergent semantic reasoning. Our goal is to enable a single end-to-end trained model to both learn to map robot observations to actions and enjoy the benefits of large-scale pretraining on language and vision-language data from the web. To this end, we propose to co-fine-tune state-of-the-art vision-language models on both robotic trajectory data and Internet-scale vision-language tasks, such as visual question answering. In contrast to other approaches, we propose a simple, general recipe to achieve this goal: in order to fit both natural language responses and robotic actions into the same format, we express the actions as text tokens and incorporate them directly into the training set of the model in the same way as natural language tokens. We refer to such category of models as vision-language-action models (VLA) and instantiate an example of such a model, which we call RT-2. Our extensive evaluation (6k evaluation trials) shows that our approach leads to performant robotic policies and enables RT-2 to obtain a range of emergent capabilities from Internet-scale training. This includes significantly improved generalization to novel objects, the ability to interpret commands not present in the robot training data (such as placing an object onto a particular number or icon), and the ability to perform rudimentary reasoning in response to user commands (such as picking up the smallest or largest object, or the one closest to another object). We further show that incorporating chain of thought reasoning allows RT-2 to perform multi-stage semantic reasoning, for example figuring out which object to pick up for use as an improvised hammer (a rock), or which type of drink is best suited for someone who is too sleepy (an energy drink).

Approach Overview

Results

Videos

Link to videos: https://robotics-transformer2.github.io/#videos