trentmkelly/LessWrong-43k huggingface dataset:
Announcing “Key Phenomena in AI Risk” (facilitated reading group)
by Nora_Ammann — 2023-05-09T00:31:10.294Z

TLDR: “Key Phenomena in AI Risk” is a 7 week-long, facilitated reading group. It is aimed at people interested in conceptual AI alignment research, in particular from fields such as philosophy, systems research, biology, cognitive and social sciences.

The program will run between July and August 2023. Sign up

here

by May 28th.

What?

The “Key Phenomena in AI risk” reading curriculum provides an extended introduction to some key ideas in AI risk, in particular risks from misdirected optimization or 'consequentialist cognition'. As such, it aims to remain largely agnostic of solution paradigms.

See

here

for a short overview of the curriculum;

here

for a more extensive summary; and

here

for the full curriculum.

This is a 7-week long program, which consists of a weekly 90’ facilitated call to discuss the week’s key phenomena and readings, as well as individual time for reading (min. 2h, more if you would like to explore the optional readings).

The courses are virtual and free of charge.

For Who?

The curriculum is primarily aimed at people interested in conceptual research in AI risk and alignment.

It is designed to be accessible to audiences in, among others, philosophy (of agency, knowledge, power, etc.) and systems research (e.g. biological, cognitive, information-theoretic, social systems, etc.).

When?

The reading groups will be taking place in July and August 2023.

We expect to run 2-6 groups à 4-8 participants (including 1 facilitator). Each group will be led by a facilitator with substantive knowledge of AI risk.

Overview of the curriculum

Week 0 is dedicated to getting to know each other and clarifying how the program will work.
Week 1 focuses on why important features of generally intelligent 'consequentialist cognition' might be algorithmically realizable, and its potential implications.
Week 2 focuses on why it can be hard to direct such intelligence in a safe and beneficial direction.
Week 3 discusses instrumental convergence in goal-oriented systems. .
Week 4 discusses risks from systems that seek predictive omniscience.
Week 5 discusses some factors on why surveilling (or oversight of) these artificial systems may be fraught with differential advantage for deceptive tendencies.
Week 6 discusses why even an incoherent aggregation of optimi

...
Yoshua Bengio argues for tool-AI and to ban "executive-AI"
by habryka — 2023-05-09T00:13:08.719Z
Link: https://yoshuabengio.org/2023/05/07/ai-scientists-safe-and-useful-ai/

Yoshua Bengio wrote a blogpost yesterday in which he argues for developing "scientist AI", which seems in-structure very similar to historical Tool-AI proposals.

For the (IMO) best response to this kind of proposal see Gwern's: Why Tool AIs Want to Be Agent AIs.

Below I copies the blogpost in-full, since all of it seems pretty relevant.

AI Scientists: Safe and Useful AI?

Published

7 May 2023

by

yoshuabengio

There have recently been lots of discussions about the risks of AI, whether in the short term with existing methods or in the longer term with advances we can anticipate. I have been very vocal about the importance of accelerating regulation, both nationally and internationally, which I think could help us mitigate issues of discrimination, bias, fake news, disinformation, etc. Other anticipated negative outcomes like shocks to job markets require changes in the social safety net and education system. The use of AI in the military, especially with lethal autonomous weapons has been a big concern for many years and clearly requires international coordination. In this post however, I would like to share my thoughts regarding the more hotly debated question of long-term risks associated with AI systems which do not yet exist, where one imagines the possibility of AI systems behaving in a way that is dangerously misaligned with human rights or even loss of control of AI systems that could become threats to humanity. A key argument is that as soon as AI systems are given goals – to satisfy our needs – they may create subgoals that are not well-aligned with what we really want and could even become dangerous for humans.

Main thesis: safe AI scientists

The bottom line of the thesis presented here is that there may be a path to build immensely useful AI systems that completely avoid the issue of AI alignment, which I call AI scientists because they are modeled after ideal scientists and do not act autonomously in the real world, only focusing on theory building and question answering. The argument is that if the AI system can provide us benefits without having to autonomously act in the world, we do not need to solve the AI alignment problem. This would suggest a policy banning powerful autonomous AI systems that can act in the world (“executives” rather than “scientists”) unless proven safe. However, such a solution would still leave open the po

...
H-JEPA might be technically alignable in a modified form
by Roman Leventov — 2023-05-08T23:04:20.951Z

This post is a collection of replies to Steven Byrnes' "LeCun’s “A Path Towards Autonomous Machine Intelligence” has an unsolved technical alignment problem" which wind up into a whole separate post because I suggest tweaks to the H-JEPA architecture (aka APTAMI) that might address the problems pointed out by Byrnes.

In the end, I also suggest a reason to ponder on the H-JEPA architecture and its safety properties even if we don't expect it to work and/or don't actually plan to deploy it. In short, this reason is to simply train ourselves, as AI alignment and cognitive scientists, to understand and find problems with AI-generated alignment science and proposals.

Below are quotes from Steven's post and my replies.

Recommended skimming strategy: read section headings and bolded sentences in the text.

The ambivalence between pro-sociality and instrumental thinking could be useful

Such an AI would feel “torn”, so to speak, when deciding whether to advance its own goals versus humans’.

This ambivalence could also be a necessary condition for useful and/or safe general intelligence. People themselves are torn between pro-social and selfish motives all the time. Totally controllable and subservient robot might turn out not to be a very useful or downright dangerous thing, e.g., in the hands of misaligned people.

This is somewhat related to the currently open question of whether a "tool"/simulator AI is safer than an "agent" AI with a certain ethical ego or vice versa, all things considered.

The Altruism Controller

We can hope that the prosocial drives will “win out” in the AI’s reckoning, but we don’t have any strong reason to expect that they will in fact win out.

Let’s say there is a submodule within the Trainable Critic module, or as a completely separate module, called Altruism Controller, where the statistics are kept (or learned, by an NN) about how much do pro-social components of intrinsic cost (or, ultimately, trainable cost) vs. “instrumental” components (power, resources, interestingness, etc.) contribute, depending on the context: the world state and the level of inference. Let’s say, on the very low levels of inference, such as precise locomotive planning and control, instrumental components of energy usually dominate, i.e., fine-grained motion planning is usually motivated by power economy and low-level instrumental expediency, such as opening a kitch...
All AGI Safety questions welcome (especially basic ones) [May 2023]
by steven0461 — 2023-05-08T22:30:50.267Z

tl;dr: Ask questions about AGI Safety as comments on this post, including ones you might otherwise worry seem dumb!

Asking beginner-level questions can be intimidating, but everyone starts out not knowing anything. If we want more people in the world who understand AGI safety, we need a place where it's accepted and encouraged to ask about the basics.

We'll be putting up monthly FAQ posts as a safe space for people to ask all the possibly-dumb questions that may have been bothering them about the whole AGI Safety discussion, but which until now they didn't feel able to ask.

It's okay to ask uninformed questions, and not worry about having done a careful search before asking.

AISafety.info - Interactive FAQ

Additionally, this will serve as a way to spread the project Rob Miles' team[1] has been working on: Stampy and his professional-looking face aisafety.info. This will provide a single point of access into AI Safety, in the form of a comprehensive interactive FAQ with lots of links to the ecosystem. We'll be using questions and answers from this thread for Stampy (under these copyright rules), so please only post if you're okay with that!

Stampy - Here to help everyone learn about stamp maximization AGI Safety!

You can help by adding questions (type your question and click "I'm asking something else") or by editing questions and answers. We welcome feedback and questions on the UI/UX, policies, etc. around Stampy, as well as pull requests to his codebase and volunteer developers to help with the conversational agent and front end that we're building.

We've got more to write before he's ready for prime time, but we think Stampy can become an excellent resource for everyone from skeptical newcomers, through people who want to learn more, right up to people who are convinced and want to know how they can best help with their skillsets.

Guidelines for Questioners:

No previous knowledge of AGI safety is required. If you want to watch a few of the Rob Miles videos, read either the WaitButWhy posts, or the The Most Important Century summary from OpenPhil's co-CEO first that's great, but it's not a prerequisite to ask a question.
Similarly, you do not need to try to find the answer yourself before asking a question (but if you want to test Stampy's in-browser tensorflow semantic search that might get you an answer quicker!).
Also feel free to ask questions that you'r

...
Predictable updating about AI risk
by Joe Carlsmith — 2023-05-08T21:53:34.730Z

(Cross-posted from my website. Podcast version here, or search "Joe Carlsmith Audio" on your podcast app.)

"This present moment used to be the unimaginable future."

- Stewart Brand

1. Introduction

Here’s a pattern you may have noticed. A new frontier AI, like GPT-4, gets released. People play with it. It’s better than the previous AIs, and many people are impressed. And as a result, many people who weren’t worried about existential risk from misaligned AI (hereafter: “AI risk”) get much more worried.[1]

Now, if these people didn’t expect AI to get so much better so soon, such a pattern can make sense. And so, too, if they got other unexpected evidence for AI risk – for example, concerned experts signing letters and quitting their jobs.

But if you’re a good Bayesian, and you currently put low probability on existential catastrophe from misaligned AI (hereafter: “AI doom”), you probably shouldn’t be able to predict that this pattern will happen to you in the future.[2] When GPT-5 comes out, for example, it probably shouldn’t be the case that your probability on doom goes up a bunch. Similarly, it probably shouldn’t be the case that if you could see, now, the sorts of AI systems we’ll have in 2030, or 2050, that you’d get a lot more worried about doom than you are now.

But I worry that we’re going to see this pattern anyway. Indeed, I’ve seen it myself. I’m working on fixing the problem. And I think we, as a collective discourse, should try to fix it, too. In particular: I think we’re in a position to predict, now, that AI is going to get a lot better in the coming years. I think we should worry, now, accordingly, without having to see these much-better AIs up close. If we do this right, then in expectation, when we confront GPT-5 (or GPT-6, or Agent-GPT-8, or Chaos-GPT-10) in the flesh, in all the concreteness and detail and not-a-game-ness of the real world, we’ll be just as scared as we are now.

This essay is about what “doing this right” looks like. In particular: part of what happens, when you meet something in the flesh, is that it “seems more real” at a gut level. So the essay is partly a reflection on the epistemology of guts: of visceral vs. abstract; “up close” vs. “far away.” My views on this have changed over the years: and in particular, I now put less weight on my gut’s (comparatively skeptical) views about doom.

But the essay is also about grokking s...
Annotated reply to Bengio's "AI Scientists: Safe and Useful AI?"
by Roman Leventov — 2023-05-08T21:26:11.374Z
Link: https://yoshuabengio.org/2023/05/07/ai-scientists-safe-and-useful-ai/

The AI safety proposal was published by Yoshua Bengio on May 7, 2023.

Below I quote the salient sections of the proposal and comment on them in turn.

Main thesis: safe AI scientists

[…] I would like to share my thoughts regarding the hotly debated question of long-term risks associated with AI systems which do not yet exist, where one imagines the possibility of AI systems behaving in a way that is dangerously misaligned with human rights or even loss of control of AI systems that could become threats to humanity. A key argument is that as soon as AI systems are given goals – to satisfy our needs – they may create subgoals that are not well-aligned with what we really want and could even become dangerous for humans.
The bottom line of the thesis presented here is that there may be a path to build immensely useful AI systems that completely avoid the issue of AI alignment, which I call AI scientists because they are modeled after ideal scientists and do not act autonomously in the real world, only focusing on theory building and question answering. The argument is that if the AI system can provide us benefits without having to autonomously act in the world, we do not need to solve the AI alignment problem. This would suggest a policy banning powerful autonomous AI systems that can act in the world (“executives” rather than “scientists”) unless proven safe. However, such a solution would still leave open the political problem of coordinating people, organizations and countries to stick to such guidelines for safe and useful AI. The good news is that current efforts to introduce AI regulation (such as the proposed bills in Canada and the EU, but see action in the US as well) are steps in the right direction.

Bengio’s “AI scientists” are similar in spirit to tool AI, Oracle AI, and the “simulators” agenda. All these ideas share a weakness: tool AI may still cause a lot of harm in the hands of “misaligned” people, or people with poor ethics. I’m not sure which AI design would be on balance better[1]: tool AI or agent AI, but my intuition, stemming perhaps from the “skin in the game” principle, tells me that agent AI would be on balance better (if some other aspects of the overall system design are also done right, which I touch upon below, in the last section).

Note that in the Gaia network architecture (Kaufmann & the Digital Gaia Team, 20...
Are healthy choices effective for improving live expectancy anymore?
by Christopher King — 2023-05-08T21:25:45.549Z

The usual way to improve your life expectancy is through a healthy lifestyle of some-sort, especially during peace time.

However, in many plausible futures, personal health now seems to have little effect:

We create aligned AGI: aligned AGI could presumably solve any health problems we accumulated due to our lifestyle choices.

We create unaligned AGI: unaligned AGI would definitely solve any health problems we accumulated due to our lifestyle choices.

Humans solve aging without AGI: health problems we accumulated probably don't matter in this scenario.

Some other existential crisis occurs.

Whereas the scenarios where health choices do effect longevity seem slim:

Humans do not invent AGI and to do not solve aging themselves and no existential catastrophes occur in your lifetime

No existential catastrophes occur, we/an AI do find a cure for aging, but medical regulations prevent the cure from being distributed widely (please no 🙏).

One important consideration to keep in mind though is that things that extend our life expectancy usually have short term health benefits as well. For example, sleep, diet, and exercise have a massive effect on energy levels. But if you're only optimizing short-term health, does the optimal lifestyle look different?
LeCun’s “A Path Towards Autonomous Machine Intelligence” has an unsolved technical alignment problem
by Steven Byrnes — 2023-05-08T19:35:19.180Z

Summary

This post is about the paper

A Path Towards Autonomous Machine Intelligence

(APTAMI) by Yann LeCun. It’s a high-level sketch of an AI architecture inspired by the brain.
APTAMI is mostly concerned with arguing that this architecture is a path towards more-capable AI. However, it is also claimed (both in the paper itself and in

associated public communication

) that this architecture is a path towards AI that is “controllable and steerable”, kind, empathetic, and so on.
I argue that APTAMI is in fact, at least possibly, a path towards that latter destination, but only if we can solve a hard and currently-unsolved technical problem.
This problem centers around the Intrinsic Cost module, which performs a role loosely analogous to “innate drives” in humans—e.g. pain being bad, sweet food being good, a curiosity drive, and so on.
APTAMI does not spell out explicitly (e.g. with pseudocode) how to create the Intrinsic Cost module. It offers some brief, vague ideas of what might go into the Intrinsic Cost module, but does not provide any detailed technical argument that an AI with such an Intrinsic Cost would be controllable / steerable, kind, empathetic, etc.
I will argue that, quite to the contrary, if we follow the vague ideas in the paper for building the Intrinsic Cost module, then there are good reasons to expect the resulting AI to be not only unmotivated by human welfare, but in fact motivated to escape human control, seek power, self-reproduce, etc., including by deceit and manipulation.
Indeed, it is an open technical problem to write down any Intrinsic Cost function (along with training environment and other design choices) for which there is a strong reason to believe that the resulting AI would be controllable and/or motivated by human welfare, while also being sufficiently competent to do the hard intellectual tasks that we’re hoping for (e.g. human-level scientific R&D).
I close by encouraging LeCun himself, his colleagues, and anyone else to try to solve this open problem. It’s technically interesting, very important, and we have all the information we need to start making progress now. I've been working on that problem myself for years, and I think I’m making more than zero progress, and if anyone reaches out to me I’d be happy to discuss the current state of the field in full detail.
…And then there’s an e

...
Product Endorsement: Apollo Neuro
by Elizabeth — 2023-05-08T19:00:02.404Z

Short version: This $310 vibrating bracelet dramatically improved my sleep and moderately improved my emotional regulation. The return policy is pretty liberal so if this seems at all appealing I recommend trying it, or one of the cheaper alternatives I haven’t investigated. Between now and Mothers’ Day they are $300.

[note: the link I use here is an affiliate link that gives you a $40 discount and me a $40 Amazon gift card, but I think it won’t apply for either of us if you buy during the sale]

Backstory

I have a pretty liberal threshold for trying shit Facebook advertises to me. Most of it is crap, but every once in a while there is something amazing that justifies all the work and return fees. Previous purchases include resistance band clips that measure force applied, sleep-safe headphones that claimed to measure your HRV, and an infrared heat massage. But when I first looked at the Apollo Neuro’s website, it was too dumb even for me. The explanation for how it worked was a mix of absent and stupid, and the rush to provide scientific evidence was somehow worse than nothing. I only tried it because a friend raved about it. He had also thought it sounded deeply stupid and only bought it because his friend raved about it, and I can only assume she also thought it was stupid until someone raved about it to her, in a great circle of life. Now it is your turn to be told it sounds stupid but it works.

[For some people. My sample size is only three people.]

Benefits to me

My sleep improved a lot. I don’t have precise metrics for this because fitbit is stupid, but 1-3 times a week I wake up feeling drugged (positive valence) because my muscles are so relaxed. This never happened before the neuro unless I took actual drugs*. I also estimate my number of remembered wake-ups has been ~halved.

The Neuro has about 8 hours of battery. When I started using it I always woke up with the battery drained, meaning I’d activated it repeatedly. I now wake up with it at 70-90% charge, partially because I wake up less and thus am not turning it back on, and partially because I use less intense vibration.

[*TBF I regularly take supplements for sleep, so what I really mean here is “unless I took more drugs than baseline”]

My emotional regulation improved as well. There were a number of stressful things I handled better than my baseline. Some of this is subjective, but th...
Acausal trade naturally results in the Nash bargaining solution
by Christopher King — 2023-05-08T18:13:09.114Z

In this post, I demonstrate that if we treat acausal trade as being analogous to ordinary trade (in the sense that it obeys supply and demand), the final result is the Nash bargaining solution.

Supply and demand of utility

Alice is an AI that wants to maximize apples. Bob is an AI that wants to maximize oranges. One day, they decide to trade.

Let's say Alice releases that by producing 3 fewer apples, she could produce 4 oranges. And Bob can produce 4 apples by forgoing 3 oranges. So Alice and Bob can achieve a better outcome if they both do this. Notice that this is, in essence, just ordinary trade. (Note that in the acausal context where Alice only believes Bob has a 50% chance of existing, the apples he produces only count for 50%. For simplicity, we will treat them as having physically met and both having a 100% chance of existing.)

How much more should Alice and Bob trade? We can imagine there being a supply and demand curve for apples, priced in oranges. Alice demands apples (by paying a price in oranges) and Bob supplies apples (by being paid a price in oranges). Where these curves meet is the optimal trade. In particular, the point consists of a price p and a number of apples x. The number of oranges in the trade will be y=px.

The two properties of utility trading

This trade has the following two properties:

The "books balance": the amount of oranges provided by Alice are equal in value to the apples provide by Bob when compared with the price p.

The solution maximizes value: the trade maximizes the value when we treat an orange as being p times as valuable as an apple. In particular, the trade is on the Pareto front. Note that it does not need be the only solution that maximizes the value. The reason the trade maximizes value is as follows: If there is a more valuable solution with more apples, than Bob's supply curve was too low (he could have offered more apples at the price p). If there is a superior solution with more oranges, the price was too high (Alice could have offered more oranges, thus lowering the price).

In practice, acausal trade is more than just trading physical objects. But I still think these two criteria are natural criteria for a satisfactory trade. If a solution gives one agent twice as many utils as the other, they should be willing to give up one of their utils to give the...