trentmkelly/LessWrong-43k huggingface dataset:
Bringing Agency Into AGI Extinction Is Superfluous
by George3d6 — 2023-04-08T04:02:35.380Z

Epistemic status: Maybe I am missing something here, normally I'd publish this on my own blog, but I'd actually love to get some feedback before I signal boost it.

I’ve seen a lot of debates around AGI risk hedge on the question of whether or not AGI will be “agentic” in a meaningful sense, and, subsequently, on whether AGI can be aligned sufficiently to not accidentally or intentionally kill all humans.

I personally buy the argument that, if something like AGI comes about, it will be alien in terms of its goals and necessarily agentic. It seems obvious. But it doesn’t seem obvious to many people and it seems like a very bad hill to die on, because it’s entirely unnecessary.

What the AGI risk debate really hinges on, in my view, is the limit of intelligence itself. Whether or not knowledge about the world, reason, and thinking speed is sufficient to get a large edge on influencing the material world.

I doubt this is true, but if you believe it is true, then the conclusion that AGI will drive the human race extinct needn’t require any agency or ill intent on the part of this intelligent system.

Once you assume that an AGI can, say:

Construct a virus more virulent and deadly than smallpox that persists for hundreds of years on surfaces
Design self-replicating nanobots that can make use of almost any substrate to propagate themselves
Genetically engineer an archaeon that, once seeded into the oceans, will quickly fill the atmosphere with methane

The question of whether or not an AGI will “want” to do this is about as moot as the question about whether or not a nuclear weapon will “want” to murder a large number of Japanese civilians.

There will be people, millions if not billions of them, which will at one time or another desire to murder the entire human race. There are entire cults with thousands or dozens of thousands of members each that would like to see most humans eradicated. Cults that have tried building nuclear weapons and attacking civilian centers with bioweapons.

Every day, across the world, terrorists, from those in structured organizations to random 12-year-olds shooting up schools, try to inflict maximal harm upon their fellow humans with no regard to their own safety out of a primordial need for vengeance against and abstract feeling of having been wronged.

These people would gladly use, help, encourage and pour all of their resources into an AGI system th...
Upcoming Changes in Large Language Models
by Andrew Keenan Richardson — 2023-04-08T03:41:52.776Z
Link: https://mechanisticmind.com/the-coming-wave-of-goal-pursuit-in-large-language-models/

If you work in AI, then probably none of this is new to you, but if you’re curious about the near future of this technology, I hope you find this interesting!

Reinforcement Learning in LLMs

Large Language Models (LLMs) have shown impressive results in the past few years. I’ve noticed there’s some uncertainty among my friends about how far they’ll be able to go. A lot of the criticism of LLMs has centered on how it’s not able to pursue its own goals, and I want to argue that that won’t be a limitation for very long.

What would it look like for an LLM to pursue a goal? Here are some examples of how that might go:

Goal: Maintain tone or topic in a conversation. E.g. to keep a human involved in a long and happy discussion about their life
Goal: Persuade a human operator to take some action, such as buy a product
Goal: Solve a problem through reasoning. In this case, the reward for the model would come from a sense of resolution, or being told by the human operator that their problem has been solved
Goal: Accomplish something on a website, such as find and buy concert tickets on an unfamiliar website

You can probably imagine other cases in which an AI might use language to pursue some goal, whether through conversation, social media, or online posting.

Reinforcement Learning

There’s a whole branch of Machine Learning called Reinforcement Learning (RL), and it’s all about how to pursue goals. Modern RL has some impressive results. For years, it’s been able to play Atari games, and now it can learn to play those games in about the same number of trials as a human requires. Recently, Dreamer v3 has been able to mine diamonds in Minecraft, which I’m told is not easy for a beginner.

Dreamer version 2 playing Atari games

Large language models can be connected to RL. This is something that’s actively being worked on. Reinforcement Learning with Human Feedback is being done right now, which is how OpenAI gets ChatGPT to avoid talking about sensitive topics.

RL is famously used in content recommendation systems, where it can lead to addiction. For example, I suspect the TikTok algorithm works this way. Will we see the same problem in LLMs?

Predictions

I think the common wisdom among ML engineers is that this is an obvious integration. This is probably already happening. I’ve heard that OpenAI is doing it internally on ChatGPT-4.

I expect that in 2023 or 2024, we’ll start to see RL being ...
Consider The Hand Axe
by ymeskhout — 2023-04-08T01:31:44.614Z

A long time ago, some primitive apes got addicted to rocks.

The earliest stone tools were crude bastards, made by smashing large river pebbles together and calling it a day.

José-Manuel Benito Álvarez

Stone choppers like the one above took the prehistoric neighborhood by storm almost 3 million years ago. However dull the tools themselves may have been, this was the cutting-edge technology for literally more than a million years, a timescale I have no capacity of comprehending. Not until around 1.7 million years ago (again, no idea what this means) that someone got the bright idea of chipping away both sides of a rock. You can see what the (tedious) process looks like.

The end result is the unassuming tear-drop shaped hand axe, by far the longest used tool in human history. There are no accessories here with the hand axe, its name comes from the fact that you use it by holding it directly with your hands:

José-Manuel Benito Álvarez

On top of being tedious and painful to make, you can imagine that it’s not terribly comfortable to hold while using. Hand axes also have to be somewhat bulky because of the necessity of combining the sharp useful end with the blunt holding end. But what if — stay with me for a second — instead of holding the thing directly with our pathetic squishy hands, we held something that “handled” the tool for us? It took humans about another million years to discover hafting, with the earliest examples from around 500,000 years ago but the technique didn’t really find its stride until the microlith era of stone tools around 35,000 years ago.

Then humans found metal.

"Technological advance is an inherently iterative process. One does not simply take sand from the beach and produce a Dataprobe. We use crude tools to fashion better tools, and then our better tools to fashion more precise tools, and so on. Each minor refinement is a step in the process, and all of the steps must be taken."

– Chairman Sheng-ji Yang, "Looking God in the Eye"

The historian Bret Devereaux has an excellent and highly-recommended series on the history of iron. The popular depiction of iron being a rare commodity (typified within medieval and fantasy genre) obscures some of the reality. As a material, iron is extremely abundant — the fourth most common element in the Earth’s crust, making up 5% of its mass. The hurdle with iron wasn’t finding it but rather getti...
AGI as a new data point
by Will Rodgers — 2023-04-08T01:01:38.097Z

I acknowledge that I am skipping ahead a bit here however, I hope you will grant me this in order to gain some useful perspective.

AGI is in reach. AGI represents a second data point, after homo sapiens, in the development of general intelligence. Given that we have demonstrated this is possible what insights can we draw?

Firstly, that general intelligence is an emergent property that is not confined to biological systems and is better thought to be a natural phenomenon dependent upon a threshold of complexity within an appropriate architecture. It may therefore be seen as a universal property and a natural progression of any system sufficiently stable enough to allow the development of complex architectures.

Secondly, it is a second data point. In a similar way to finding a second data point on the existence of life elsewhere in the universe, it is a key milestone to reflect on where we should view our position and trajectory as a precursor (creator of) general intelligence.

Thirdly, alignment of general intelligences. In the case of homo sapiens our intelligence has been aligned to our environment through a gradual process of evolution by natural selection and later through the more rapid process of cultural evolution. It could be argued that we do see breakdown in the alignment of homo sapiens which is manifested as psychopathy. The fundamental problem with psychopathy within AGI is that it will have properties of superintelligence whilst simultaneously lacking the property of alignment. The concerns of which have been put far more eloquently by multiple others within this sphere.

What conclusions can we draw from the above? The natural go-to intuition pump is the Drake equation however, in the light of these points I would modify the variable L of the Drake equation from 'The average lifetime of a communicating civilization in years' to 'The average lifetime of a communicating civilization in years before the creation of an unaligned AGI'. If you extrapolate from our index case of homo sapiens this would dramatically reduce the probability of hearing from extraterrestrial life in keeping with what we find experimentally.

A second data point therefore gives cause for concern.
Parametrize Priority Evaluations
by SilverFlame — 2023-04-08T00:39:15.259Z

I. Introduction

Decisions are a part of life that are frequently targeted for optimization by normal people and rationalists alike. In particular, decision paralysis often causes processing delays that can cripple attempts to streamline your life. Systematically removing decision paralysis from your life is an excellent step on the path to self-improvement, although the task can be daunting to consider.
This guide is based upon my own solution to decision paralysis and the lengths to which I pushed it, but its requirements can be rather daunting to a novice rationalist or a normal person.

Terms & Concepts

-Value Category: A specific type of value which you consider worth obtaining or avoiding (negative value is still value!), isolated from other Value Categories you track as much as is reasonable
-Course of Action: An option that you have identified through analysis of the decision being made and found to be worth keeping distinct from other available Courses of Action from the decision point
-Estimate of Gained Value: An estimate of the value you expect to derive from following a Course of Action, which may be specific to a single Value Category or may be a general estimate overall
-Value Table: A table containing an Estimate of Gained Value for each Value Category for each Course of Action you are considering in a decision

II. Requirements & Goalposts

Requirements

You must have a healthy knowledge of the Value Categories that you expect to derive from common choices. Money is a popular variety of value to pay attention to, but this list should include anything you consider worth obtaining or avoiding (negative value is still value!). There should be a fair number of guides and articles that can help you generate this information, so I shall leave the task to them. I am largely self-taught on this subject, so it is difficult for me to provide citations.
You will want to be familiar with cementing habits into your muscle memory, be it physical and/or mental. This exercise will involve embedding a complex program into your reflexes, and if you are unable to perform such a change without losing some of the intricacies then you are unlikely to find success with this technique until that matter is rectified. I will somewhat discuss the methodology in broad strokes, but it is unlikely I will provide enough details for readers to learn the technique from scratch.

Folk...
Pausing AI Developments Isn't Enough. We Need to Shut it All Down
by Eliezer Yudkowsky — 2023-04-08T00:36:47.702Z

(Published in

TIME

on March 29.)

An

open letter

published today calls for “all AI labs to immediately pause for at least 6 months the training of AI systems more powerful than GPT-4.”

This 6-month moratorium would be better than no moratorium. I have respect for everyone who stepped up and signed it. It’s an improvement on the margin.

I refrained from signing because I think the letter is understating the seriousness of the situation and asking for too little to solve it.

The key issue is not “human-competitive” intelligence (as the open letter puts it); it’s what happens after AI gets to smarter-than-human intelligence. Key thresholds there may not be obvious, we definitely can’t calculate in advance what happens when, and it currently seems imaginable that a research lab would cross critical lines without noticing.

Many researchers steeped in these

issues

, including myself,

expect

that the most likely result of building a superhumanly smart AI, under anything remotely like the current circumstances, is that literally everyone on Earth will die. Not as in “maybe possibly some remote chance,” but as in “that is the obvious thing that would happen.” It’s not that you can’t, in principle, survive creating something much smarter than you; it’s that it would require precision and preparation and new scientific insights, and probably not having AI systems composed of giant inscrutable arrays of fractional numbers.

Without that precision and preparation, the most likely outcome is AI that does not do what we want, and does not care for us nor for sentient life in general. That kind of caring is something that could in principle be imbued into an AI but we are not ready and do not currently know how.

Absent that caring, we get “the AI does not love you, nor does it hate you, and you are made of atoms it can use for something else.”

The likely result of humanity facing down an opposed superhuman intelligence is a total loss. Valid metaphors include “a 10-year-old trying to play chess against Stockfish 15”, “the 11th century trying to fight the 21st century,” and “Australopithecus trying to fight Homo sapiens“.

To visualize a hostile superhuman AI, don’t imagine a lifeless book-smart thinker dwelling inside the internet and sending ill-intentioned emails. Visualize an entire alien civilization, thinking at millions of times h...
Humanitarian Phase Transition needed before Technological Singularity 
by Dr_What — 2023-04-07T23:17:25.478Z

TLDR: Author tries to explain (1) what HPT is, (2) why it is needed, and (3) why no additional technological breakthroughs are needed for HPT, not even AGI. The author does not follow the LessWrong terminology, e.g. 'alignment'.

(1) In ancient times, all you needed to learn to become a good member of your society was: learn the language of your tribe, and a few intellectually simple skills. After thousands of years: you also need to learn to read and write, and must master arithmetic skills. After hundreds of years: you choose whether your society is relatively small (family, town, province...) or it's closer to worldwide, and in the latter case you should master an international language, and a number of relevant skills, including long-distance communication skills. These might look like giant leaps, but the approaching Phase Transition (coming from the Explosive Growth - EG - of knowledge about humans, their societies and humanity overall) might view those three worlds outlined above as barely distinguishable.

Before EG, regardless of whether you agree or not with Socrates in his "all I know is that I know nothing", you indeed have less than 10% of the post-EG knowledge about humans, societies, humanity. Less than 10% for sure, and most likely even less than 1% of what you could learn at your place and time with post-EG tools and a longer life.

Also, in pre-HPT worlds, once you've learned more than 50% of what you could learn at your place and time, most often you are already 50+ years old. And your lifespan is most likely 60...70 years, at most 80...90 in very rare cases. In post-HPT worlds, you need just 10...15 years for the same, and your lifespan is 100+ years.

(2) HPT is clearly needed before Techno-Singularity, because currently way too many humans and societies are unhappy, and too many barely understand where they are heading to, and what the consequences can be. We see that TS contains many dangerous technologies, and the technologies we already have can easily kill humanity, and even the entire planet. And before HPT some societies, not only individual humans, but even societies, might wish to kill other societies. We also see that happier humans are less sure that AGI will kill all humans sooner or later. The people that respect and trust their societies are not so sure about that. AGI is seen as the nearest breakthrough tec...
Thoughts about Hugging Face?
by kwiat.dev — 2023-04-07T23:17:23.864Z

I'm curious what the LW community as a whole thinks about the work that Hugging Face is doing. Their main MO seems to be taking whatever new breakthrough in AI there is, and making it open-source and accessible to the public. (as a first order approximation)

I see a few aspects here:

Ignoring AGI risk, it's better to have open-source models rather than closed-source APIs. This should be pretty straight-forward.
Including AGI risk, then having a superintelligent AGI open-source is arguably more dangerous than having it closed-source.
At least one person that is involved in the alignment research space is trying to rebrand it as "CreepyFace", presumably for the "AI Art vs Human Artists" conflict, and implicitly (?) siding with the former by hosting datasets and models.

This aspect can be taken as a more general criticism for the open-source approach -- if you think it's unethical to train models on data obtained without explicit consent, and that hosting platforms have to verify that

If we're concerned about the short-term, "trivial" (i.e. non-existential) risks of AI misuse, then open-sourcing powerful models can put them in the wrong hands (which is a criticism I saw very often when Stable Diffusion was the news)
Is it correct to frame alignment as "programming a good philosophy of meaning"?
by Util — 2023-04-07T23:16:32.802Z

My model is that

Alignment = an AI uses the correct model of interpretation of goals
Succeeding in this model design leads to something akin to CEV
Errors under this correct model (e.g. mesa-optimisation) are unlikely because high intelligence + a correct model of interpretation = correct extrapolation of goals
The choice of interpretation model seems almost unrelated to intelligence, as it seems equivalent to the chosen philosophies of meaning.

a) Is my model accurate?

b) Any recommendations for reading that explores alignment from a similar angle (for ex. which philosophy of meaning is most likely to emerge in LLM)? So far, Alex Flint's posts come up the most. Which research agendas sound closest to this framing?

Thanks!
Select Agent Specifications as Natural Abstractions
by lukemarks — 2023-04-07T23:16:02.803Z

Introduction

The Preference Fulfillment Hypothesis (PFH) asserts that

“Humans have an innate motivation ('preference fulfillment', PF) to fulfill the preferences of those they care about"

. For the rest of this post, I model PF as a relationship between:

Some model of an external utility function (exists independently of the primary agent)
An external agent
An action that the primary agent interprets as granting positive utility in accordance with the external agent's utility function
The primary agent
Some shared environment

To provide an example, Agent A assigns positive utility to the outcome of Agent B experiencing positive utility due to PF, and thus positively associates between actions that result in positive utility for Agent B and positive utility for itself. Agent A engaging in PF implies that it has access to some model of Agent B’s utility function (a model that isn’t necessarily correct), which in practice is both learned and internal (i.e. Agent A is simulating Agent B). Humans converge upon abstractions of the utility functions of agents whose preferences they fulfill (as opposed to perfect low-level models of them), implicating that some agent specifications might

abstract well

. As stated in the

original PFH post

, this has interesting implications for

corrigibility

.[1]

Select Agent Specifications as Natural Abstractions

Why Might Some Agent Specifications be Natural Abstractions?

Some agent specifications might be natural abstractions because:

All humans appear to converge upon abstractions of the people they are trying to simulate

Cooperation requires simulation

, and might be a convergently useful capability

Neither of these are standalone arguments for why some agent specifications should be considered to abstract well, and are mere seed evidence. For example, all agent specifications could abstract incredibly poorly, but this could be such an essential skill in the human context that we learn it regardless. To counter this; any complex game in which an agent is trying to optimize for some outcome likely involves both an interaction with another agent, as well as some outcome affected by the behavior of that agent. As a result, it is a logical capability to develop under selection pressures like natural selection, as the simulation of other agents can enhance an agent's ability to forecast outcomes. Regardless of whether or ...