trentmkelly/LessWrong-43k huggingface dataset:
Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover
by Ajeya Cotra — 2022-07-18T19:06:14.670Z

I think that

in the coming 15-30 years

, the world could plausibly develop “transformative AI”: AI powerful enough to bring us into a new, qualitatively different future, via

an explosion in science and technology R&D

. This sort of AI

could

be sufficient to make this the

most important century of all time for humanity

.

The most straightforward vision for developing transformative AI that I can imagine working with very little innovation in techniques is what I’ll call human feedback[1] on diverse tasks (HFDT):

Train a powerful

neural network

model to simultaneously master a wide variety of challenging tasks (e.g. software development, novel-writing, game play, forecasting, etc) by using reinforcement learning on

human feedback

and other metrics of performance.

HFDT is not the only approach to developing transformative AI,[2] and it may not work at all.[3] But I take it very seriously, and I’m aware of increasingly many executives and ML researchers at AI companies who believe something within this space could work soon.

Unfortunately, I think that if AI companies race forward training increasingly powerful models using HFDT, this is likely to eventually lead to a

full-blown AI takeover

(i.e. a possibly

violent uprising or coup

by AI systems). I don’t think this is a certainty, but it looks like the best-guess default absent specific efforts to prevent it.

More specifically, I will argue in this post that humanity is more likely than not to be taken over by misaligned AI if the following three simplifying assumptions all hold:

The “racing forward” assumption:

AI companies will aggressively attempt to train the most powerful and world-changing models that they can, without “pausing” progress before the point when these models could

defeat all of humanity combined

if they were so inclined.

The “HFDT scales far” assumption:

If HFDT is used to train larger and larger models on more and harder tasks, this will eventually result in models that can autonomously advance frontier science and technology R&D, and continue to get even more powerful beyond that; this doesn’t require changing the high-level training strategy itself, only the size of the model and the nature of the tasks.

The “naive safety effort” assumption:

AI companies put substantial effort into ensuring their mode

...
Turning Some Inconsistent Preferences into Consistent Ones
by niplav — 2022-07-18T18:40:02.243Z

cross-posted from niplav.github.io

Epistemic Status

This is still a draft that I was told to already post here, which includes working (but very slow) code for one special case. Hopefully I'll be able to expand on this in the next ~half year.

Representing inconsistent preferences with specific mathematical
structures can clarify thoughts about how to make those preferences
consistent while only minimally changing them. This is discussed in
the case of preferences over world states, represented by directed
graphs; and preferences
over lotteries
of world states, represented either by infinitely dense
graphs, (in some cases) vector fields over probability
simplices, or edge-weighted directed graphs. I also present
an algorithm for the discrete case based on the graph edit
distance. Implications
for scenarios such as ontological
shifts are discussed.

Turning Some Inconsistent Preferences into Consistent Ones

A kind of God-made (or evolution-created) fairness between species is
also unexpectedly found.

— Yew-Kwang Ng, “Towards Welfare Biology: Evolutionary Economics of Animal Consciousness and Suffering” p. 1, 1995

Random testing is simple in concept, often easy to implement, has been
demonstrated to effectively detect failures, is good at exercising systems
in unexpected ways (which may not occur to a human tester), and may be
the only practical choice when the source code and the specifications
are unavailable or incomplete.

— Tsong Yueh Chen/Fei-Ching Kuo/Robert G. Merkel/T.H. Tse, “Adaptive Random Testing: the ART of Test Case Diversity”, 2010

Consider an agent which displays (von
Neumman-Morgenstern)
inconsistent preferences,
for example choosing two incompatible
options in the two scenarios in the Allais
paradox, or reliably
displaying cycles
in its actions (detecting which actions are in fact caused by
inconsistent preferences, and not just exotic ones from weird
abstractions, is considered a separate problem here). We might want
to interact with that agent, e.g. trade with it, help it (or exploit
it), or generally know how it will act But how to go about that if the
agent displays inconsistent preferences? Perhaps it might even be the
case that humans are such agents, and find ourselves in a conundrum:
we know our preferences are inconsistent and reliably exploitable,
and that agents with such preferences reliably fare worse in the
world,
we might wa...
Addendum: A non-magical explanation of Jeffrey Epstein
by lc — 2022-07-18T17:40:37.099Z

Around seven months ago I wrote the post "a non-magical explanation of Jeffrey Epstein". In it, I make the argument that Jeffrey Epstein did actually kill himself and that he was a CIA agent. I reach that conclusion mostly by drawing on some Conspiracy Theory and doing some light research.

I am mostly satisfied with my reasoning, and in particular my conclusion that Epstein was a high level informant for American intelligence services. However, over time I became less satisfied with my conclusion about Epstein's death. The paradox of the Epstein case is that the circumstances of his death seem so organized, and yet succeeding investigative details and a nuanced understanding of the parties involved seem to completely rule out foul play. It's quite literally a locked room murder story.

Problems with the murder hypothesis

Impracticality, lack of forensic evidence, lack of witnesses

Multiple reviewers of footage from that night, including the Attorney General at the time, claimed related cameras around Epstein's cell showed no one exiting or entering the block he was in. I am satisfied in the assumption that not all of the parties who verified this made a mistake or are secretly controlled by some puppet-master behind the scenes. This means Epstein would have to be killed by someone in his housing block who escaped their cell during the night, broke into Epstein's cell without any witnesses, and then strangled him to death without leaving any obvious and unmistakable evidence of a struggle for the coroner, and this seems totally implausible. Epstein didn't yell as his neighboring inmate sat there picking the lock? Get blood and skin under his fingernails? Do literally anything while this degenerate ninja-inmate tied his bedsheets around his neck? None of the other inmates thought to mention this if that's what happened, even after it was clear there was a coverup and they might be able to swing immunity out of it?

Epstein's will & foreknowledge of his death

Epstein clearly knew when he was going to die, because he drafted and signed a will just two days before he did. You can make up murder stories where he knows this, but Epstein seems like the type of person to cause maximum damage to his killers by explicitly telling others who he believed was going to off him and why.

Lack of viable suspects

I don't think Epstein's sex abuse victims have accused anyone wi...
Launching a new progress institute, seeking a CEO
by jasoncrawford — 2022-07-18T16:58:00.777Z
Link: https://rootsofprogress.org/seeking-a-ceo

Summary: The Roots of Progress is planning a major expansion of our activities, and we are seeking a Chief Executive Officer to lead the new organization in partnership with me (I will remain Founder & President). We’re taking this step because we see an opportunity to do much more for the progress movement, going well beyond my essays and talks. Our initial focus will be on a “career accelerator” for public intellectuals in progress studies.

Our mission and why it matters

The progress of the last few centuries—in science, technology, industry, and the economy—is one of the greatest achievements of humanity. But progress is not automatic or inevitable. We must understand its causes, so that we can keep it going, and even accelerate it.

But in order to make progress, we must believe that progress is possible and desirable. The 19th century believed in the power of technology and industry to better humanity, but in the 20th century, this belief gave way to skepticism and distrust. We can’t go back to the naive views of the past, but we need a new way forward.

Our mission is to establish a new philosophy of progress for the 21st century and beyond—one based on the ideas of humanism and agency, and one that puts forth a bold, ambitious vision for the technological future.

Our opportunity

Last year, I announced that this blog was becoming a one-man nonprofit research organization (after starting it as an intellectual side project in 2017, and becoming a full-time independent researcher in 2019). Since then, it has become clear that there is too much energy and support for this mission—and too much to do!—for this organization to remain focused solely on my own research and writing.

The new philosophy of progress needs a movement to establish it. The pillars of this movement are:

Intellectual foundations: a lot of research, thinking, and writing, to better understand and communicate the lessons of progress, and to apply them to the problems of today and the opportunities for tomorrow.
Community-building: events, forums, meetups, and conferences for the progress community to exchange ideas, forge relationships, and start projects.
Cultural outreach: from school curricula, to inventor biopics, to sci-fi that paints a positive vision of the future.

Our strategy and programs

Of these three pillars, intellectual foundations are fundamental. Our initial focus will be on creatin...
Machine Learning Model Sizes and the Parameter Gap [abridged]
by Pablo Villalobos — 2022-07-18T16:51:22.132Z
Link: https://epochai.org/blog/machine-learning-model-sizes-and-the-parameter-gap

Summary: The model size of notable Machine Learning systems has grown ten times faster than before since 2018. After 2020 growth has not been entirely continuous: there was a jump of one order of magnitude which persists until today. This is relevant for forecasting model size and thus AI capabilities.

Trends in model size

In current ML systems, model size (number of parameters) is related to performance via known

scaling laws

. We used

our dataset

to analyze trends in the model size of 237 milestone machine learning systems. The systems are categorized into Language, Vision, Games and Other according to the task they solve.

Model size slowly increased by 7 orders of magnitude from the 1950s to around 2018. Since 2018, growth has accelerated for language models, with model size increasing by another 4 orders of magnitude in the four years from 2018 to 2022 (see Figure 1). Other domains like vision have grown at a more moderate pace, but still faster than before 2018.

Figure 1. Top: Transition period around 2018, assuming a single post-2018 trend. Bottom: the same period, assuming two separate post-2018 trends.

The parameter gap

Starting in 2020, we see many models below 20B parameters and above 70B parameters, but very few in the 20B-70B range. We refer to this scarcity as the parameter gap (see Figure 2).

Figure 2: Model size over time, separated by domain. Red lines highlight the parameter gap. Most systems above the gap are language or multimodal models.

We have come up with some hypotheses that explain the parameter gap, of which these two are the ones most consistent with the evidence:

Increasing model size beyond 20B parameters has a high marginal cost due to the need to adopt different parallelism techniques, so that mid-sized models are less cost-effective than bigger or smaller ones.
GPT-3 initiated the gap by ‘jumping’ one order of magnitude in size over previous systems. This gap was maintained because researchers are incentivized to build the cheapest model that can outperform previous models. Those competing with GPT-3 are above the gap; the rest are below.

The existence of the parameter gap suggests that model size has some underlying constraints that might cause discontinuities in the future.

Read the full paper now on the arXiv
Quantilizers and Generative Models
by Adam Jermyn — 2022-07-18T16:32:33.944Z

Thanks to Evan Hubinger for discussions about quantilizers, and to James Lucassen for discussions about conditioned generative models. Many of these ideas are discussed in Jessica Taylor's Quantilizers: A Safer Alternative to Maximizers for Limited Optimization: this post just expands on a particular thread of ideas in that paper. Throughout I’ll refer to sections of the paper. I have some remaining confusion about the “targeted impact” section, and would appreciate clarifications/corrections!

Abstract

This post explores the relationship between quantilizers and generative models. My main takeaways are:

A natural way to build a quantilizer is by sampling from an appropriately-conditioned generative model.
Unfortunately quantilizing doesn’t seem to confer much advantage over the underlying generative model: to the extent that a quantilizer is more powerful than a generative model, it’s more dangerous, and vice versa.
Quantilizing is pretty computationally expensive relative to the advantage it brings, making it unclear if this is a competitive approach even if it conferred a net safety advantage at fixed power.

Definitions

I’ll follow the setup in “Quantilizers: A Safer Alternative to Maximizers for Limited Optimization” (Section 1, Definition 1) and say that a quantilizer is a model with:

A base distribution γ, which describes a set of outputs we think are “normal” or “unlikely to lead to doom”.
A quantile q∈[0,1].
An ordering over outputs (e.g. a utility or loss function).

A quantilizer returns a random output from the top q fraction of outputs from γ according to the ordering. So for instance a 0-quantilizer is a utility maximizer, a 0.1-quantilizer returns outputs in the best 10%, etc.

Where does the base distribution come from?

One challenge with quantilizers is that they require a base distribution γ that we trust. That is, we want to be able to sample from γ many times and never hit bad outputs. Unfortunately, we don’t know how to explicitly write this distribution down.

A natural alternative to explicit construction is to train a generative model on the distribution of actions a humans takes (Section 3.1). Then for any given task γ can be written as

γtask=P(action|good human attempts task)

This seems quite doable. Train GPT-N on text written by humans, then prompt it with the task:

Prompt: You

...
AI Hiroshima (Does A Vivid Example Of Destruction Forestall Apocalypse?)
by Sable — 2022-07-18T12:06:31.955Z

After Eliezer's post, I got to thinking about how to think about AI destroying the world, and how such a fate might be averted. I claim no special expertise, and an epistemic status of great uncertainty.

Humanity has faced (and so far, survived) one potential apocalypse already: that of nuclear war. I refer specifically to the Cold War, where the United States and Russia came very close to launching nuclear weapons against each other several times.

It occurred to me that, by the beginning of the Cold War, the world had a concrete example of the kind of destruction that nuclear weapons were capable of: the annihilation of two Japanese cities at the end of World War II, namely Hiroshima and Nagasaki.

In other words, while there were likely some unknowns, there was no confusion or ambiguity about what nuclear weapons were capable of, or that they represented an incredibly dangerous technology. While the technology could be and was harnessed for a variety of purposes, no one was under any illusions about the potential downsides.

If they needed a reminder, they had only to look at the Human Shadow of Death.

Nuclear War was (and continues to be) a legitimate threat to human civilization.[1] And yet it hasn't happened yet, and we appear to be past the worst period of risk.

So what lessons might we learn from this, and how might they apply to AI?

The question that motivated this post was simple:

Did the destruction of Hiroshima and Nagasaki make a nuclear war between the United States and Russia more or less likely?

In other words, if World War II had come to an end some other way, and no nuclear bomb had ever been detonated in anger - would the Cold War have been more likely to turn hot?

I believe so, without having researched the topic extensively. In a world with no vivid demonstrations of the horror of nuclear weaponry, how could it not be easier to imagine using them?

How could the examples of Hiroshima and Nagasaki make someone more willing to inflict the same devastation upon other cities, and face that devastation deployed against their own population centers?

I believe, based solely on priors, that the vivid examples of the danger of nuclear warfare helped avert a nuclear apocalypse.

So what about AI?

One of the problems acknowledged by those who seek to align AGI is that the dangers of unaligned intelligences are not salient to politicians, decision-makers, or the ...
How the ---- did Feynman Get Here !?
by George3d6 — 2022-07-18T09:43:49.482Z
Link: https://www.epistem.ink/p/how-the-fuck-did-feynman-get-here

How The Fuck Did Feynman Get Here !?I’ve been re-rereading stories about Feynman again. People decry that no such geniuses exist today, they oft seem to decry the fact that no physicists since has developed such a thirst for understanding and beautiful ability to abstract for others.

But the question, I think, lies better on its head:

How the fuck did someone like Richard Feynman ever become a physicist?

It often seems that Feynman’s most impressive skills are perpendicular to anything relevant to physics, mathematics, or even science beyond a very abstract philosophical level.

For example, his approaches to solving complex problems often sound like the kind of thing you’d do before figuring out how to optimally reason using the efficient abstractions provided by mathematics:

”I can’t understand anything in general unless I’m carrying along in my mind a specific example and watching it go. Some people think in the beginning that I’m kind of slow and I don’t understand the problem, because I ask a lot of these “dumb” questions: “Is a cathode plus or minus? Is an an-ion this way, or that way?” But later, when the guy’s in the middle of a bunch of equations, he’ll say something and I’ll say, “Wait a minute! There’s an error! That can’t be right!” The guy looks at his equations, and sure enough, after a while, he finds the mistake and wonders, “How the hell did this guy, who hardly understood at the beginning, find that mistake in the mess of all these equations?” He thinks I’m following the steps mathematically, but that’s not what I’m doing. I have the specific, physical example of what he’s trying to analyze, and I know from instinct and experience the properties of the thing. So when the equation says it should behave so-and-so, and I know that’s the wrong way around, I jump up and say, “Wait! There’s a mistake!”

This approach to problems is impressive, but it’s impressive in the same way that a man doing complex calculations without a computer is. It seems tedious beyond reason for someone able to abstract better.

Mind you, if Feynman was a businessman or even an engineer this would be a rather useful way of thinking and it is the way that the vast majority of “applied” people I know seem to approach dialogues with their more theoretically minded colleagues.

There’s also this interesting sorry, though I’m afraid I can’t be arsed to cite it, where Feynman found a fa...
Conditioning Generative Models for Alignment
by Jozdien — 2022-07-18T07:11:46.369Z

This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the

Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program

. It builds on work done on simulator theory by Janus, who came up with the strategy this post aims to analyze; I’m also grateful to them for their comments and their mentorship during the AI Safety Camp, to Johannes Treutlein for his feedback and helpful discussion, and to Paul Colognese for his thoughts.

TL;DR: Generative models don’t operate on preferences in the way agents or optimizers do, instead just

modelling a prior over the world that can return worlds satisfying some conditional, weighted by how likely they are given this prior

. This implies that these models may be safer to use than more standard optimizers, although we would likely use them in different ways. For example, at the limit of capability we could use it to do alignment research by simulating a world with a superhuman alignment researcher. That isn’t to say that these models are without their own problems however, carrying with them versions of the outer and inner alignment problems that are subtly different from those of optimizers. This post explores how generative models are different, ways in which we might use them for alignment, some of the potential problems that could come up, and initial attempts at solving them.

How do generative models work?

Generative models are trained by self-supervised learning to model a distribution of data. With large language models, the real distribution underlying the textual data comprising its training corpus represents the world as it is now, in theory. As they get stronger, we can assume that they’re getting better at modelling this world prior, as I’ll describe it going forward.

We have the ability to prompt these models, which, at a low-level, means that we can give it the first part of some text that it has to complete. High-level, this means that we can describe some property of the world that may or may not exist, and the model samples from the world prior and uses a weighted distribution of the likely worlds that satisfy that property (for simplicity at the cost of technical accuracy, you can imagine it simulating the most likely world satisfying the property). For example, if you give it a prompt that says an asteroid is about to impact the world and finishes wi...