trentmkelly/LessWrong-43k huggingface dataset:
The replication and emulation of GPT-3
by Ben Cottier — 2023-01-16T01:40:25.148Z

This post is one part of the sequence Understanding the diffusion of large language models. As context for this post, I strongly recommend reading at least the 5-minute summary of the sequence.

Key takeaways

Estimated quantity	GPT-3 (May 2020)	OPT-175B (May 2022)
Actual compute cost of the final training run	$12M (90% CI: $5M–$33M) (

reasoning

)

$1.7M (90% CI: $1.5M–$2.3M)

(

reasoning

)

Actual total compute cost (inc. trial and error, preliminary experiments)[1]	$49M (90% CI: $15M–$130M) (

calculation

)	$6M (90% CI: $4M–$10M) (

calculation

)
Core team size[2]	25 people	11 people
Project duration[3]	92 days	78 days

Table 1: comparison of resources required for GPT-3 and OPT-175B, which was intended to replicate GPT-3.

It took about 23 months until an actor that explicitly aimed to replicate GPT-3 succeeded and published about it (namely, Meta AI Research publishing

OPT-175B

in May 2022). The only other explicit replication attempt I am aware of has not succeeded; this is the GPT-NeoX project by the independent research collective EleutherAI. Other actors produced GPT-3-like models as early as May 2021 (12 months after GPT-3’s publication) and throughout the subsequent year. These were not explicit replication attempts, but they are evidence of which actors are capable of replication. (

more

)
In Table 1, I estimate the compute cost, talent, and time required to produce and replicate GPT-3. To estimate the requirement for replication, I used OPT-175B as the canonical example. (

more

)
In addition to the above estimates, I find that following

Hoffmann scaling laws

leads to a roughly 3x decrease in compute cost to produce a GPT-3 equivalent, compared to using the same model size and training tokens as the original GPT-3. (more)

In addition to reducing compute, Hoffmann scaling laws predict a much smaller model size (I estimate 15 billion parameters) to match GPT-3 performance, which reduces the engineering talent requirement and the inference cost.
Given this reduced cost, actors like EleutherAI seem capable of replicating GPT-3 now (December 2022), as long as they can collect a large enough dataset to train effectively on ~1 trillion tokens (which seems like a significant but much smaller obstacle than the compute and engineering requirements).

The above estimates assume a fixed set of circumstances. In reality, comp

...
GPT-3-like models are now much easier to access and deploy than to develop
by Ben Cottier — 2023-01-16T01:39:26.088Z

This post is one part of the sequence Understanding the diffusion of large language models. As context for this post, I strongly recommend reading at least the 5-minute summary of the sequence.

EDIT 25-Feb-2023: I have made a big update from the claims in this post about deployment of large language models costing less than development in total. I now think that for the largest, most commercially successful LLMs, the total compute spent on deployment is much larger than in development. However, this doesn't significantly update my conclusion about the importance of focusing on development rather than deployment as a target of intervention (point 2c in the Key Takeaways). More information in this comment.

Key takeaways

GPT-3 itself can be used and fine-tuned via an API. Despite this, there’s still demand for direct access to the model’s weights, and multiple similar models now exist that provide access to weights. OPT-175B is a GPT-3 replica; I estimate that its model weights can be downloaded by hundreds to thousands of ML researchers in academia, government, and industry labs, subject to approval. BLOOM is similar to GPT-3, but not a replica, and is publicly available for anyone to download. (

more

)
What resources are required to actually use these models? (

more

)

One relevant question is how much money and talent it would take to generate a large volume of text with about the same average usefulness as GPT-3’s outputs. (

more

)

Based on publicly listed pricing, running GPT-3 via the OpenAI API would generate approximately 150 million English words for $4000.[1]
I estimate that a user could generate 150 million English words per day of similar usefulness to GPT-3’s outputs for as little as $240 per day by running one instance of BLOOM independently (i.e., downloading the model weights and running the model on a server that they either directly rent or own).
I estimate there are 5000 people (90% CI: 100 to 45,000) that are capable of running BLOOM independently.[2]

What about for a very large-scale application of a GPT-3-like model—for example, generating text equivalent to 1% of global Twitter activity for one year, or assisting one million software developers with coding for one year? I estimate that deploying a model like BLOOM in these ways would be 20% of the cost of developing the model (90% CI: 10 to 68%), in terms of the dollar cost of compute al

...
Background for "Understanding the diffusion of large language models"
by Ben Cottier — 2023-01-16T01:38:56.473Z

This post is one part of the sequence Understanding the diffusion of large language models. As context for this post, I strongly recommend reading at least the 5-minute summary of the sequence.

Definitions

Here I explain what various terms mean in the context of the diffusion of AI technology and this sequence.

Transformative AI (TAI)

I define transformative AI (TAI) qualitatively as “potential future AI that precipitates a transition comparable to (or more significant than) the agricultural or industrial revolution.”

This is how Holden Karnofsky originally defined the term in 2016 (Karnofsky, 2016). Luke Muehlhauser elaborated on the definition in 2019, emphasizing that “the transition immediately following the (‘first’) industrial revolution was a much larger change in [measures] of human empowerment and well-being than other events in recorded history, including the Black Death of the 14th century, the Scientific Revolution of the 17th century, the two world wars of the 20th century, and the ‘2nd and 3rd industrial revolutions’ implied by discussions of a potential future ‘4th industrial revolution’” (Muehlhauser, 2019).

Artifact: a product of AI R&D

An artifact is a product of AI research or development. Examples of artifacts include a trained model, the code for training a model, a new dataset, or a new algorithm. Artifacts are what diffuse among actors.

I use the term “tangible” to contrast artifacts with tacit or procedural knowledge (e.g. the intuition that some research idea is worth trying). It can be useful to consider the diffusion of tacit or procedural knowledge, but in this sequence I focus on artifacts.

Diffusion: the spread of artifacts among actors

Diffusion is when an actor acquires an existing artifact. For example, suppose a machine learning model is trained by an AI lab and the model is made freely available to download from the internet. The model may then diffuse rapidly and widely to any individual that is able to download and run the model on their computer.

Note that the meaning of “acquires an existing artifact” is deliberately vague. An AI technology being acquired by some actor does not necessarily mean that they have acquired the exact same object (e.g., the exact same ML model), but it means that the actor possesses an object that serves roughly the same function as an existing object.
“Diffus

...
Understanding the diffusion of large language models: summary
by Ben Cottier — 2023-01-16T01:37:40.416Z

5-minute summary

Time from GPT-3’s publication (May 2020) to…	Time	Date	Model	Organizations involved
A better model is produced[1]	7 months[2]	Dec 2020	Gopher	Deepmind

A better or equally good model is open-sourced[3]

AND

A successful, explicit attempt at replicating GPT-3 is completed

23 months

(equally good model[4])

May 2022	OPT-175B	Meta AI Research

Table 1: Key facts about the diffusion of GPT-3-like models

I present my findings from case studies on the diffusion of nine language models that are similar to OpenAI’s GPT-3 model, including GPT-3 itself.[5] By “diffusion”, I mean the spread of artifacts among different actors, where artifacts include trained models, code, datasets, and algorithmic insights.[6] Diffusion can occur through different mechanisms, including open publication and replication.[7] Seven of the models in my case studies are “GPT-3-like” according to my definition, which basically means they are similar to GPT-3 in design and purpose, and have similar or better capabilities. Two models have clearly worse capabilities but were of interest for other reasons. (more)
I think the most important effects of diffusion are effects on (1) AI timelines—the leading AI developer can get to TAI sooner by using knowledge shared by other developers, (2) who leads AI development, (3) by what margin they lead, and (4) how many actors will plausibly be contenders to develop transformative AI (TAI)[8]. The latter three effects in turn affect AI timelines and the competitiveness of AI development. Understanding cases of diffusion today may improve our ability to predict and manage the effects of diffusion in the lead up to TAI being developed. (more)
See Table 1 for key facts about the timing of GPT-3-like model diffusion. Additionally, I’m 90% confident that no model exists which is (a) uncontroversially better than GPT-3 and (b) has its model weights immediately available for download by anyone on the internet (as at November 15, 2022). However, GLM-130B (Tsinghua University, 2022)—publicized in August 2022 and developed by Tsinghua University and the Chinese AI startup Zhipu.AI—comes very close to meeting these criteria: it is probably better than GPT-3,[9] but still requires approval to download the weights. (more)
I’m 85% confident that in the two years since the publication of GPT-3 (in May 2020), publicly known GPT-

...
Speculation on Path-Dependance in Large Language Models.
by NickyP — 2023-01-15T20:42:48.186Z

Epistemic Status: Highly Speculative. I spent less than a day thinking about this in particular, and though I have spent a few months studying large language models, I have never trained a language model. I am likely wrong about many things. I have not seen research on this, so it may be useful for for someone to do a real deep dive.

Thanks to Anthony from the Center on Long Term Risk for sparking the discussion earlier today for this post. Also thanks to conversations with Evan Hubinger ~1 year ago that got me thinking about the topic previously.

Summary

My vague suspicions at the moment are somewhat along the lines of:

Training an initial model: moderate to low path-dependance
Running a model: high "prompt-dependance"
Reinforcement Learning a Model: moderate to high path-dependance.

Definitions of "low" and "high" seem somewhat arbitrary, but I guess what I mean is how different behaviours of the model can be. I expect some aspects to be quite path dependant, and others not so much. This is trying to quantify how many aspects might have path-dependance based on vibe.

Introduction

Path dependence is thinking about the "butterfly effect" for machine learning models. For highly path-dependant models, small changes in how a model is trained can lead to big differences in how it performs. If a model is highly path-dependant, then if we want to understand how our model will behave and make sure it's doing what we want, we need to pay attention to the nitty-gritty details of the training process, like the order in which it's learning things, or the random weights initialisation. And, if we want to influence the final outcome, we have to intervene early on in the training process.

I think having an understanding of path-dependance is likely useful, but have not really seen any empirical results on the topic. I think that in general, it is likely to depend on different training methods a lot, and in this post I will give some vague impressions I have on the path dependance of Large Language Models (LLMs).

In this case, I will also include "prompt-dependance" as another form of "path-dependance" when it comes to the actual outputs of the models, though this is not technically correct since it does not depend on the actual training of the model.

Initial Training of a Model

My Understanding: Low to Moderate Path-Dependance

So with Large Language Models at the moment, the main way t...
Underspecification of Oracle AI
by Rubi J. Hudson — 2023-01-15T20:10:42.190Z

Rubi and Johannes worked on this post as part of the SERI MATS program, with Evan Hubinger providing mentorship to both. Rubi also received mentorship from Leo Gao. Thanks to Paul Colognese and Nicholas Schiefer for discussions related to this post.

An oracle is a type of AI system that only answers questions without taking any other actions in the world. Simulators and generative models, which have seen increased discussion recently (links:

1

,

2

,

3

,

4

), can be thought of as types of oracles. Such systems may simultaneously be powerful enough to generate a pivotal act while also being easier to align due to a more limited domain.

One

major concern

with oracles is that the answers they give can still manipulate the world. If oracles are evaluated on predictive accuracy, this gives them an incentive to use their answers to affect the course of events and make the world more predictable. Concretely, we are concerned that oracles may make self-fulfilling prophecies (also known as self-confirming predictions or fixed points), where the act of making the prediction causes the predicted outcome to come true. Even if their manipulation

does not terminate in a fixed point

, attempts to influence the world towards predictability can be very dangerous.

As one example, consider a highly trusted oracle asked to predict the stock market. If such an oracle predicts that stock prices will rise, then people buy based off that prediction and the price will in fact rise. Similarly, if the oracle predicts that prices will fall, then people will sell, causing prices to fall. For a more real world example, see

this market

and

this market

, each on whether a specific person will find a research/romantic partner. Here, high probabilities would indicate desirability of that person, while low probabilities would suggest some hidden flaw, either of which could influence whether potential partners decide to reach out and therefore how the market resolves.

In both the stock market and partnership cases, multiple predictions are valid, so how does the oracle choose between them? Ideally, we would like it to choose the one that is “better” for humanity, but this now introduces an outer alignment question similar to an agentic AI acting directly on the world, and which we wanted to avoid by using oracles in the first place.

Instead, what we can aim for is a...
How Does the Human Brain Compare to Deep Learning on Sample Efficiency? 
by DragonGod — 2023-01-15T19:49:33.301Z

I have an impression that within lifetime human learning is orders of magnitude more sample efficient than large language models, but there are numerous caveats to this:

We don't have "an ecological evaluation objective" for language models (they weren't actually optimised for the downstream language usage tasks on which we compare them to humans)

Insomuch as we do have an ecological evaluation objective (predictive loss on the test set) language models are already very superhuman and apparently even GPT-1 was superhuman at next token prediction

Though for similar reasons, next token prediction is not an ecological training objective for humans

Humans that specialised at next token prediction (the way some humans specialise at chess) mat show markedly different results

It's plausible that most of the optimisation involved in producing the brain happened over the course of our evolutionary history and within lifetime human learning is more analogous to fine tuning than to training from scratch.

#3 notwithstanding, I'm curious if we have any robust estimates for how within lifetime human learning compares to deep learning on sample efficiency across various tasks of interest.

Why Does This Matter?

The brain is known to be very energy efficient compared to GPUs of comparable processing power.

However, energy efficiency is just much less taut of a constraint for human engineering than it was for biology (electricity has a much higher throughput than ATP and we have a much larger energy budget). This relative energy abundance would likely remain the case (or rather intensify) as AI systems become more capable.

Thus, the energy efficiency of the brain does not provide much evidence with respect to whether advanced AGI will be neuromorphic.

On the other hand, it seems very plausible that data efficiency is just part and parcel of general intelligence. It may be the case that sufficiently powerful systems would necessarily be more data efficient than the brain (this seems very plausible to me).

If deep learning is sufficiently less data efficient than the brain, it may provide evidence that deep learning wouldn't produce existentially dangerous systems.

We may thus have reason not to expect deep learning to scale to superhuman general intelligence.
Deceptive failures short of full catastrophe.
by Alex Lawsen  — 2023-01-15T19:28:52.531Z

Epistemic status: trying to unpack a fuzzy mess of related concepts in my head into something a bit cleaner.

A lot of my concern about risks from advanced AI is because of the possibility of

deceptive alignment

. Deceptive alignment has already been discussed in detail on this forum, so I don’t intend to re-tread well worn ground here, go read what Evan’s written. I’m also worried about a bundle of other stuff that I’m going to write about here, all of which can loosely be thought of as related to deception, but which don’t fit e.g. the definition of deceptive alignment from the Risks from Learned Optimization paper. The overall cluster of ideas might be loosely summarised as ‘things which cause oversight to fail but don’t look like a misaligned planner ‘deliberately’ causing overseers to believe false things’.

Importantly, I think that most of the things I want to describe can still occur under a relaxing of the conditions required for the precise definition of deceptive alignment. I think the issues l describe below are less concerning than deceptive alignment, but they still pose a significant risk, as they may undermine oversight processes and give false reassurance to human overseers. Many, if not all, of the ideas below aren’t particularly new, though a couple might be, and the clustering might be useful for others. While my motivation for writing this is mostly to try to clean up ideas in my own head, I do somewhat frequently see different kinds of deception confused for or conflated with each other, and think there’s value in being precise about what we mean, especially when it comes to discussing evidence of deceptive behaviour in current or near-future systems.

I’ve very loosely grouped the deceptive failures I want to discuss into ‘camouflage’ failures and ‘inattention’ failures. To give some initial intuition:

Camouflage failures are failures associated with the difficulty of incentivising “don’t do X” without also incentivising “don’t let me see you do X”.
Inattention failures are intended to highlight the difference between “I’m not trying to deceive you” and “I’m trying not to deceive you”.

Camouflage

There are many cases where a model behaves ‘deceptively’, in the sense that it is easy to draw incorrect conclusions about its behaviour, because this behaviour is directly rewarded during training. This means camouflage can occ...
Non-directed conceptual founding 
by TsviBT — 2023-01-15T14:56:36.940Z

[Metadata: crossposted from https://tsvibt.blogspot.com/2022/08/non-directed-conceptual-founding.html. Written 13 June 2022. I'm fairly likely to not respond to comments promptly. If you're especially interested in chatting, my gmail address is: tsvibtcontact ]

In trying to understand minds-in-general, we sometimes ask questions that talk about "big" things (taking "big" to ambiguously mean any of large, complex, abstract, vague, important, touches many things, applies to many contexts, "high-level"). E.g.:

What is it for a mind to have thoughts or to care about stuff? How does care and thought relate?

What is it to believe a proposition?

Why do agents use abstractions?

These "big" things such as thought, caring, propositions, beliefs, agents, abstractions, and so on, have to be analyzed and re-understood in clearer terms in order to get anywhere useful. When others make statements about these things, I'm pulled to pause their flow of thoughts and instead try to get clear on meanings. In part, that pull is because the more your thoughts use descriptions that aren't founded on words with clear meaning, the more leeway is given to your words to point at different things in different instances.[1]

Main claim

From talking with Sam, I've come to think that there's an important thing I hadn't seen sufficiently clearly:

A description of Y that uses terms that are only as "foundational" as Y or even "less foundational" than Y, can still be useful and doesn't have to be harmful. For analyzing "big" things, such descriptions are necessary.

Circular founding

A description is a proposition of the form "Y is a ...". A description is founded on X if it assumes that X exists, e.g. by mentioning X, or by mentioning Z which mentions X, or by relying on X to be in the background.[2]

Some descriptions of Y might be founded on Y, or on X where X is itself founded on Y. A description like that could be called circular, or in general non-directed.

The circularity could be harmful. E.g., you could trick yourself into thinking you're talking about anything coherently, when really you're not: whenever you ask "Wait, what's Y?" you respond "Oh it's XZ", and you say "Z is YX", and you say "X is YZ", and you never do the work of connecting XYZ to stuff that matters, so it's all hot air. Or, you might have "Y" more densely connected to its neighbors, but not beholden to anything ...