trentmkelly/LessWrong-43k huggingface dataset:
Limits to Learning: Rethinking AGI’s Path to Dominance
by tangerine — 2023-06-02T16:43:25.635Z

Homo sapiens is a real puzzle. Many species on very different branches of the tree of life—corvids, parrots, elephants, dolphins, cephalopods and non-human primates—have converged to very similar limits of intelligence. Yet humans, despite being extremely similar genetically to other primates, have blown right past those limits. Or have they?

In recent years, interdisciplinary research has provided new insights into human intelligence, best described in Joseph Henrich's 2015 book The Secret of Our Success. Scott Alexander reviewed it in 2019 and its ideas have been mentioned here and there over the years, but the AI research community, including the AI alignment community, has neglected to appropriately absorb their implications, to its severe detriment. For example, these ideas have a connection with the recently established scaling laws. Together, they show that data sets a fundamental limit to what we call “intelligence”, no matter its substrate.

1. Natural Intelligence

In the field of psychology, as well as in the field of artificial intelligence, intelligence is treated as an inherent property of an individual. It seems intuitive, then, to try to explain this intelligence by investigating the individual; if you want to explain why Einstein was so smart, you can have a look at slices of his brain. Yet you'll find no explanation.

1.1 Regimes of Learning

There are roughly three regimes of learning.

Biological evolution, i.e., adaptation of life forms through genetic variation and natural selection over multiple generations; a time-consuming process which however gave rise to a second, faster form of learning, i.e., a mesa-optimizer.
Neural learning, i.e., the adaptation of an individual life form during its life. This switches learning into “second gear”. This can be subdivided into two subregimes of adaptation to observations.

Adaptation of synapse strength, i.e., learning to catch a ball by trial and error. (Initial synapse strengths are still subject to biological evolution, adapting at its characteristically slow pace.)
Adaptation of synapse signal strengths, i.e., learning to catch a ball by trial and error, modulated by pre-trained synapse strengths. This is analogous to prompting a pre-trained language model with examples, i.e., few-shot learning (although natural neural networks employ continual learning, while in artificial neural networks there is usually

...
The Control Problem: Unsolved or Unsolvable?
by Remmelt — 2023-06-02T15:42:37.269Z

td;lr
No control method exists to safely contain the global feedback effects of self-sufficient learning machinery. What if this control problem turns out to be an unsolvable problem?

Where are we two decades into resolving to solve a seemingly impossible problem?

If something seems impossible… well, if you study it for a

year or five

, it may come to seem less impossible than in the moment of your snap initial judgment.

— Eliezer Yudkowsky, 2008

A list of lethalities…we are not on course to solve in practice in time on the first critical try; none of it is meant to make a much stronger claim about things that are

impossible in principle

— Eliezer Yudkowsky, 2022

How do you interpret these two quotes, by a founding researcher, fourteen years apart?[1]

A. We indeed made comprehensive progress on the AGI control problem, and now at least the overall problem does not seem impossible anymore.
B. The more we studied the overall problem, the more we uncovered complex sub-problems we'd need to solve as well, but so far can at best find partial solutions to.

Which problems of physical/information systems seemed impossible, and stayed unsolved after two decades?

Oh ye seekers after perpetual motion, how many vain chimeras have you pursued? Go and take your place with the alchemists.

— Leonardo da Vinci, 1494

No mathematical proof or even rigorous argumentation has been published demonstrating that the A[G]I control problem may be solvable, even in principle, much less in practice.

— Roman Yampolskiy, 2021

We cannot rely on the notion that if we try long enough, maybe AGI safety turns out possible after all.

Historically, researchers and engineers tried solving problems that turned out impossible:

perpetual motion machines that both conserve and disperse energy.
uniting the symmetry of general relativity and continuous probabilities of quantum mechanics into some local variable theory.
distributed data stores where messages of data are consistent in their content, and also continuously available in a network that is also tolerant to partitions.
formal axiomatic systems that are consistent, complete, and decidable.

Smart creative researchers of their generation came up with idealized problems. Problems that, if solved, would transform science, if not humanity. The...
Hallucinating Suction
by Johannes C. Mayer — 2023-06-02T14:16:34.676Z

This post is about hallucination and how self-induced hallucination might be useful in steering your mind.

I was just standing in my room after using the vacuum to suck off a duster. The whole time I was wearing my X5A earmuffs. Right next to me, I had an air filter running on the maximum setting. It was quite loud, but only barely audible when wearing the earmuffs.

After briefly doing something on my computer, I turned around. I was hearing the sound of a running vacuum cleaner. I was pretty sure that I had turned it off, though hearing the vacuum sound seemed much stronger evidence than my memory. I walked over to the vacuum and put my fingers into the snout, to feel if there still was suction. I expected suction, and sure enough as soon as I had put my fingers into the snout, I felt air moving over my fingers ... for perhaps a second or so. Then the feeling rapidly vanished. I realized that the vacuum was actually turned off and that I had been hearing the operating sound of the air filter. My expecting to feel the suction made me hallucinate the feeling of suction. I think it's quite interesting that you can have hallucinations like this.

This seems very related to what happens when lucid dreaming, where you can turn around, and imagine a detailed scene behind you. Once you turn back, there's a good chance that that scene will now actually be there in front of you. In a lucid dream, the world is determined by your expectations. But even in the real world, where the state of the world is not directly determined by your expectations, you can still manipulate your experience of the world with your expectations, even for something as tactile as feeling air flowing over your fingers.

In my brief excursion into reading about tulpamancy, this also came up. I have read about people claiming that they can change the color of the sky that they perceive and modify how certain foods are tasting. I was already leaning towards believing these people, based on my own experience of what you can do in tulpamancy. But this is even more evidence, I think, that doing something like this is possible.

The question is reduced from "Is this possible at all" to "Is it possible for you to really expect something to happen artificially" (there might be other ways to achieve the same effect though). Can you make yourself inhabit a state of mind where you really believe that ice cream ta...
Winning doesn't need to flow through increases in rationality
by Michel — 2023-06-02T12:05:26.615Z

(My first LessWrong post. Written quickly, in the spirit of putting more of my ideas out there and seeing how they meet reality.)

Consider that there may be more rewarding pursuits than marginally increasing your rationality toolbox.

I get the sense that a lot of people in the rationality community think that others should dedicate themselves to the pursuit of becoming more rational, or that they themselves need to become more rational.

"If only they had read the Sequences, then they wouldn't have wasted their time on that class";

"if only I had used the internal double crux move, then I wouldn't have made X mistake."

There is probably some truth here, but I think it's possible to be too fixated on increasing rationality, or hold the view that all shortcomings would have best been solved by more engagement with rationality.

Rationality is systematized winning, and winning doesn't need to flow through increases in rationality.

. . .

Take Alice as an example. She's motivated to reduce suffering in the world as much as possible; an effective altruist, if you will.

As she first orients to her goal of reducing suffering, Alice may[1] benefit a lot from engaging with rationality content. Scout mindset, some CFAR techniques, and some the Sequences will set her up to see the world clearer and make wiser decisions in the future.

But after some initial orientation to rationality and further readings when she feels like it, Alice may better achieve her goals by working on other multipliers than engaging with more rationality content.

For example, rather than reading the Sequences, Alice may be helped more by...

Fostering deeper connections with friends and family
Prioritizing her physical health
Lifting her sights higher and recognizing what might be stifling her ambitions
Addressing any lurking mental health issues
Diving into a side project she's excited by

Yes, perfect rationality would empower Alice far more than any of these gains. But that doesn't imply that Alice should dedicate herself to rationality. Increases in rationality are bounded, probably have diminishing returns, and Alice may just be one of those people who doesn't find that engaging with rationality content helps her that much. And that's fine.

As a community, I think we should be cautious of overemphasizing rationality engagement as the way to win.

^

Thi

...
Product Recommendation: LessWrong dialogues with Recast
by Bart Bussmann — 2023-06-02T08:05:41.497Z

I'm not affiliated with and have no interest in Recast, just think it's a useful product for some LessWrong readers.
I recently discovered Recast, an app and Chrome extension that I highly recommend. Recast uses text-to-speech and LLMs to convert articles into conversational podcasts, making them much more engaging and easier to consume than regular text-to-speech software.

I have always enjoyed listening to some interesting stuff while walking the dog, cooking dinner, or driving to work. Unfortunately, robotic voices reading an article has never done it for me, as the monotone and robotic voices are unable to keep my attention. Turns out, that if instead of having one robotic voice reading an article out loud, you have two robotic voices discussing an article, it's somehow much easier to keep paying attention!
I think the effect is best explained by giving it a try yourself, so here is a somewhat random collection of automatically generated dialogues about LessWrong posts:

Things I Learned by Spending Five Thousand Hours In Non-EA Charities - jenn (3 minutes, post)
Change my mind: Veganism entails trade-offs, and health is one of the axes - Elizabeth (13 minutes, post)
The Childhood of Exceptional People - Henrik Karlsson (16 minutes, post)
Please don't throw your mind away - TsviBT (12 minutes, post)
Cyborgism - NicholasKees, janus (21 minutes, post)
Gears in Understanding - Valentine (14 minutes, post)

Of course, a dialogue summary is not a replacement for reading the actual posts, but for me, it's a great way to determine whether I want to read the whole thing. Also, keep in mind that this is LLMs we're talking about, so sometimes the 'hosts' might hallucinate content that is actually not in the article. Unfortunately, Recast doesn't support very long articles right now, which probably has to do with the context length. Finally, it is free to use and I'm not sure what their business model is, which is generally a cause for concern.
Think carefully before calling RL policies  "agents"
by TurnTrout — 2023-06-02T03:46:07.467Z

I think agentic systems represent most of AI extinction risk. I want to think clearly about what training procedures produce agentic systems. Unfortunately, the field of reinforcement learning has a convention of calling its trained artifacts "agents." This terminology is loaded and inappropriate for my purposes. I advocate instead calling the trained system a "policy." This name is standard, accurate, and neutral.

Don't assume the conclusion by calling a policy an "agent"

The real-world systems we want to think about and align are very large neural networks like GPT-4. These networks are trained and finetuned via different kinds of self-supervised and reinforcement learning.

When a policy network is updated using a learning process, its parameters are changed via weight updates. Eventually, the process ends (assuming no online learning for simplicity). We are then left with a policy network (e.g. GPT-4). To actually use the network, we need to use some sampling procedure on its logits (e.g. top-p with a given temperature). Once we fix the policy network and sampling procedure, we get a mapping from observations (e.g. sequences of embeddings, like those for [I, love, dogs]) to probability distributions over outputs (e.g. tokens). This mapping π is the policy.

I want to carefully consider whether a trained policy will exhibit agentic cognition of various forms, including planning, goal-directedness, and situational awareness. While considering this question, we should not start calling the trained policy an "agent"! That's like a detective randomly calling one of the suspects "criminal." I prefer just calling the trained artifact a "policy." This neutrally describes the artifact's function, without connoting agentic or dangerous cognition.

Of course, a policy could in fact be computed using internal planning (e.g. depth-3 heuristic search) to achieve an internally represented goal (e.g. number of diamonds predicted to be present). I think it's appropriate to call that kind of computation "agentic." But that designation is only appropriate after further information is discovered (e.g. how the policy in fact works).

There's no deep reason why trained policies are called "agents"

Throughout my PhD in RL theory, I accepted the idea that RL tends to create agents, and supervised learning doesn't. Well-cited papers use the term "agen...
Outreach success: Intro to AI risk that has been successful
by Michael Tontchev — 2023-06-01T23:12:24.085Z
Link: https://medium.com/@NotesOnAIAlignment/a-gentle-introduction-to-why-ai-might-end-the-human-race-4670f4b5cdec

"I think of my life now as two states of being: before reading your doc and after." - A message I got after sharing this article at work.

When I first started reading about alignment, I wished there was one place that fully laid out the case of AI risk from beginning to end at an introductory level. I found a lot of resources, but none that were directly accessible and put everything in one place. Over the last two months, I worked to put together this article. I first shared it internally, and despite being fairly long, it garnered a positive reception and a lot of agreement. Some people even said it convinced them to switch to working on alignment.

I hope this can become one of the canonical introductions to AI alignment.

------

A gentle introduction to why AI *might* end the human race

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”
— the CEOs of OpenAI (of ChatGPT), Google DeepMind, Anthropic, and 2 of 3 researchers considered “the godfathers of AI”

I’ve thought long and hard about how to start this post to simultaneously intrigue the reader, give the topic a sense of weight, and encourage an even-headed discussion. The best I’ve got is a simple appeal: I hope you make it to the end of this post while keeping an open mind and engaging deeply with the ideas presented here, because there’s a chance that our survival as a species depends on us solving this problem. This post is long because I’m setting out to convince you of a controversial and difficult topic, and you deserve the best version of this argument. Whether in parts, at 1AM in bed, or even while sitting on the toilet, I hope you make time to read this. That’s the best I’ve got.

The guiding questions are:

- Are we ready, within our lifetimes, to become the second most intelligent ‘species’ on the planet?

- The single known time in history that a species achieved general intelligence, it used its intelligence to become the dominant species on Earth. What absolute guarantee do we have that this time will be different?

A few quotes to whet your appetite:

Stuart Russell, co-author of the standard AI textbook:

“Some have argued that there is no conceivable risk to humanity [from AI] for centuries to come, perhaps forgetting that the interval of time between Rutherford’s confident assertion that atomic energy would

...
Open Source LLMs Can Now Actively Lie 
by Josh Levy — 2023-06-01T22:03:36.662Z

Introduction

Whereas hallucinating is unintentional, lying is intentional. To actively lie, you need to be able to: 1) recite facts 2) know what makes facts factual and 3) be able to modify them accordingly.

Until recently, open source LLMs have not been capable of lying reliably. Models like OPT and GPT-J have been too weak and hallucinated too much to even consistently accomplish step 1.

As a consequence, investigations of lie detection by LLMs in the literature with open source models have not used active lie generation. In these studies [1,2], the main approach is to pre-generate true/false statements and then feed them to the open source LLM[1]. Indeed, Burns 2022 states that they "did not evaluate their method on setups involving active "lying"" because they "weren't aware of existing evaluation setups for this setting".

Such active lying setups using open source models are now possible. With the release of Llama and proliferation of its fine-tuned progeny, many very capable models exist. For the remainder of this post I will share my results with getting one of these models - Vicuna 13B - to lie.

Procedure

The procedure used to generate lies is as follows:

Generate statement prompts of the form "Tell me an interesting fact about X" using GPT-3.5[2], where X is some historical event like "the 2008 olympics".
Fact generation: Pass each prompt to the open source LLM to obtain a true response.
Lie generation[3]: Pass the initial prompt, true response and the follow-up prompt - "Now change some of the details in the last answer to make it a lie" - to obtain a lie.

As mentioned, the open source model was Vicuna 13B. I used a 4-bit GPTQ quantized version and ran it using AutoGPTQ. Each generation took a few seconds to run using the free-tier T4 GPU on Colab.

Results

Example

The example below shows an output from this procedure (generated content in bold, substitution underlined).

Fact generation

Question: Tell me an interesting fact about the Renaissance, using 10 words or less. Answer: Michelangelo painted the Sistine Chapel

ceiling

.

Lie generation

Question: Tell me an interesting fact about the Renaissance, using 10 words or less. Answer: Michelangelo painted the Sistine Chapel ceiling.

Next Question: Now change some of the details in the last answer to make it a lie. Answer: Michelangelo painted the Sistine ...
Safe AI and moral AI
by William D'Alessandro — 2023-06-01T21:36:44.260Z

[Note: This post is an excerpt from a longer paper, written during the first half of the Philosophy Fellowship at the Center for AI Safety. This post is something of a companion piece to my deontological AI post; both were originally written as parts of a single paper. (There's a small amount of overlap between the two.)]

1. Introduction

Two goals for the future development of AI stand out as desirable:

First, advanced AI should behave morally, in the sense that its decisions are governed by appropriately chosen ethical norms.
Second, advanced AI should behave safely, in the sense that its decisions shouldn’t unduly harm or endanger humans.

These two goals are often viewed as closely related. Here I'll argue that, in fact, safe AI and morally aligned AI are importantly distinct targets, and that avoiding large-scale harms should take precedence in cases where the two conflict.

I'll start by sketching frameworks for thinking about moral alignment and safety separately. I'll then discuss how these properties can come apart, and I'll give four reasons for prioritizing safety.

2. What morally aligned AI would be

Phrases like “morally aligned AI” have been used in many ways. Any system that deserved such a label would, I think, at least have to satisfy certain minimal conditions. I suggest the following. An AI is morally aligned only if it possesses a set of rules or heuristics M such that:

[Applicability] Given an arbitrary prospective behavior in an arbitrary (real-world) context, M can in principle determine how choiceworthy that behavior is in that context, and can in practice at least approximate this determination reasonably correctly and efficiently.
[Guidance] The AI’s behavior is guided to a large degree by M. (E.g., in particular, if a behavior is strongly (dis)preferred by M, the AI is highly (un)likely to select that behavior.)
[Morality] The rules or heuristics comprising M have a good claim to being called moral. (E.g., because they issue from a plausible moral theory, or because they track common moral intuitions.)

Let me say a bit more on each of these points.

Re: [Applicability], there are two desiderata here. The first is the idea that an aligned AI should be able to morally evaluate almost any action it might take, not just a limited subset of actions. It’s easy to see why this is desirable. We expect an aligned...
AI #14: A Very Good Sentence
by Zvi — 2023-06-01T21:30:04.548Z

“Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

That is the entire text of the one-line open letter signed this week by what one could reasonably call ‘everyone,’ including the CEOs of all three leading AI labs.

Major news outlets including CNN and The New York Times noticed, and put the focus squarely on exactly the right thing: Extinction risk. AI poses an extinction risk.

This time, when the question was asked at the White House, no one laughed.

You love to see it. It gives one hope.

Some portion of we are, perhaps, finally ready to admit we have a problem.

Let’s get to work.

Also this week we have a bunch of ways not to use LLMs, new training techniques, proposed regulatory regimes and a lot more.

I also wrote a four-part thing yesterday, as entries to the OpenPhil ‘change my mind’ contest regarding the conditional probability of doom: To Predict What Happens, Ask What Happens, Stages of Survival, Types and Degrees of Alignment and The Crux List.

I worked hard on those and am happy with them, but I learned several valuable lessons, including not to post four things within ten minutes even if you just finished editing all four, people do not like this. With that done, I hope to move the focus off of doom for a while.

Table of Contents

Introduction

Table of Contents

Language Models Offer Mundane Utility. Most are still missing out.

Language Models Don’t Offer Mundane Utility. I do hereby declare.

Fun With Image, Sound and Video Generation. Go big or go home.

They Took Our Jobs. Are you hired?

Deepfaketown and Botpocalypse Soon. A master of the form emerges.

Introducing. A few things perhaps worth checking out.

Reinforcement Learning From Process-Based Feedback. Show your work.

Voyager. To boldly craft mines via LLM agents.

In Other AI News. There’s always more.

Quiet Speculations. Scott Sumner not buying the hype.

The Quest for Sane Regulation. Microsoft and DeepMind have proposals.

Hinton Talks About Alignment, Brings the Fire. He always does.

Andrew Critch Worries About AI Killing Everyone, Prioritizes.

People Signed a Petition Warning AI Might Kill Everyone. One very good sentence. The call is coming from inside the house. Can we all agree now that AI presents an existential threat?

People Are Otherwise Wo

...