Spaces:
Running
Running
0:00 | |
[MUSIC] | |
0:07 | |
ANNOUNCER: Please welcome AI researcher and founding member of OpenAI, Andrej Karpathy. | |
0:21 | |
ANDREJ KARPATHY: Hi, everyone. I'm happy to be here to tell you about the state of GPT and more generally about | |
0:28 | |
the rapidly growing ecosystem of large language models. I would like to partition the talk into two parts. | |
0:35 | |
In the first part, I would like to tell you about how we train GPT Assistance, and then in the second part, | |
0:40 | |
we're going to take a look at how we can use these assistants effectively for your applications. | |
0:46 | |
First, let's take a look at the emerging recipe for how to train these assistants and keep in mind that this is all very new and still rapidly evolving, | |
0:53 | |
but so far, the recipe looks something like this. Now, this is a complicated slide, I'm going to go through it piece by | |
GPT Assistant training pipeline | |
0:59 | |
piece, but roughly speaking, we have four major stages, pretraining, | |
1:04 | |
supervised finetuning, reward modeling, reinforcement learning, and they follow each other serially. | |
1:09 | |
Now, in each stage, we have a dataset that powers that stage. We have an algorithm that for our purposes will be | |
1:17 | |
a objective and over for training the neural network, and then we have a resulting model, | |
1:23 | |
and then there are some notes on the bottom. The first stage we're going to start with as the pretraining stage. Now, this stage is special in this diagram, | |
1:31 | |
and this diagram is not to scale because this stage is where all of the computational work basically happens. This is 99 percent of the training | |
1:38 | |
compute time and also flops. This is where we are dealing with | |
1:44 | |
Internet scale datasets with thousands of GPUs in the supercomputer and also months of training potentially. | |
1:51 | |
The other three stages are finetuning stages that are much more along the lines of small few number of GPUs and hours or days. | |
1:59 | |
Let's take a look at the pretraining stage to achieve a base model. First, we are going to gather a large amount of data. | |
Data collection | |
2:07 | |
Here's an example of what we call a data mixture that comes from this paper that was released by | |
2:13 | |
Meta where they released this LLaMA based model. Now, you can see roughly the datasets that | |
2:18 | |
enter into these collections. We have CommonCrawl, which is a web scrape, C4, which is also CommonCrawl, | |
2:25 | |
and then some high quality datasets as well. For example, GitHub, Wikipedia, Books, Archives, Stock Exchange and so on. | |
2:31 | |
These are all mixed up together, and then they are sampled according to some given proportions, | |
2:36 | |
and that forms the training set for the GPT. Now before we can actually train on this data, | |
2:43 | |
we need to go through one more preprocessing step, and that is tokenization. This is basically a translation of | |
2:48 | |
the raw text that we scrape from the Internet into sequences of integers because | |
2:53 | |
that's the native representation over which GPTs function. Now, this is a lossless translation | |
3:00 | |
between pieces of texts and tokens and integers, and there are a number of algorithms for the stage. | |
3:05 | |
Typically, for example, you could use something like byte pair encoding, which iteratively merges text chunks | |
3:11 | |
and groups them into tokens. Here, I'm showing some example chunks of these tokens, | |
3:16 | |
and then this is the raw integer sequence that will actually feed into a transformer. Now, here I'm showing | |
2 example models | |
3:23 | |
two examples for hybrid parameters that govern this stage. | |
3:28 | |
GPT-4, we did not release too much information about how it was trained and so on, I'm using GPT-3s numbers, | |
3:33 | |
but GPT-3 is of course a little bit old by now, about three years ago. But LLaMA is a fairly recent model from Meta. | |
3:40 | |
These are roughly the orders of magnitude that we're dealing with when we're doing pretraining. The vocabulary size is usually a couple 10,000 tokens. | |
3:48 | |
The context length is usually something like 2,000, 4,000, or nowadays even 100,000, | |
3:53 | |
and this governs the maximum number of integers that the GPT will look at when it's trying to | |
3:58 | |
predict the next integer in a sequence. You can see that roughly the number of parameters say, | |
4:04 | |
65 billion for LLaMA. Now, even though LLaMA has only 65B parameters compared to GPP-3s 175 billion parameters, | |
4:11 | |
LLaMA is a significantly more powerful model, and intuitively, that's because the model is trained for significantly longer. | |
4:17 | |
In this case, 1.4 trillion tokens, instead of 300 billion tokens. You shouldn't judge the power of a model by | |
4:23 | |
the number of parameters that it contains. Below, I'm showing some tables of rough hyperparameters that typically | |
4:31 | |
go into specifying the transformer neural network, the number of heads, the dimension size, number of layers, | |
4:36 | |
and so on, and on the bottom I'm showing some training hyperparameters. For example, to train the 65B model, | |
4:44 | |
Meta used 2,000 GPUs, roughly 21 days of training and a roughly several million dollars. | |
4:52 | |
That's the rough orders of magnitude that you should have in mind for the pre-training stage. | |
4:57 | |
Now, when we're actually pre-training, what happens? Roughly speaking, we are going to take our tokens, | |
5:03 | |
and we're going to lay them out into data batches. We have these arrays that will feed into the transformer, | |
5:09 | |
and these arrays are B, the batch size and these are all independent examples stocked up in rows and B by T, | |
5:16 | |
T being the maximum context length. In my picture I only have 10 the context lengths, so this could be 2,000, 4,000, etc. | |
5:23 | |
These are extremely long rows. What we do is we take these documents, and we pack them into rows, | |
5:28 | |
and we delimit them with these special end of texts tokens, basically telling the transformer where a new document begins. | |
5:35 | |
Here, I have a few examples of documents and then I stretch them out into this input. | |
5:41 | |
Now, we're going to feed all of these numbers into transformer. Let me just focus on a single particular cell, | |
5:49 | |
but the same thing will happen at every cell in this diagram. Let's look at the green cell. The green cell is going to take | |
5:56 | |
a look at all of the tokens before it, so all of the tokens in yellow, and we're going to feed that entire context | |
6:03 | |
into the transforming neural network, and the transformer is going to try to predict the next token in | |
6:08 | |
a sequence, in this case in red. Now the transformer, I don't have too much time to, unfortunately, go into the full details of this | |
6:14 | |
neural network architecture is just a large blob of neural net stuff for our purposes, and it's got several, | |
6:20 | |
10 billion parameters typically or something like that. Of course, as I tune these parameters, you're getting slightly different predicted distributions | |
6:26 | |
for every single one of these cells. For example, if our vocabulary size is 50,257 tokens, | |
6:34 | |
then we're going to have that many numbers because we need to specify a probability distribution for what comes next. | |
6:40 | |
Basically, we have a probability for whatever may follow. Now, in this specific example, for this specific cell, | |
6:45 | |
513 will come next, and so we can use this as a source of supervision to update our transformers weights. | |
6:51 | |
We're applying this basically on every single cell in the parallel, and we keep swapping batches, and we're trying to get the transformer to make | |
6:58 | |
the correct predictions over what token comes next in a sequence. Let me show you more concretely what this looks | |
7:03 | |
like when you train one of these models. This is actually coming from the New York Times, and they trained a small GPT on Shakespeare. | |
7:11 | |
Here's a small snippet of Shakespeare, and they train their GPT on it. Now, in the beginning, at initialization, | |
7:17 | |
the GPT starts with completely random weights. You're getting completely random outputs as well. But over time, as you train the GPT longer and longer, | |
7:26 | |
you are getting more and more coherent and consistent samples from the model, | |
7:31 | |
and the way you sample from it, of course, is you predict what comes next, you sample from that distribution and | |
7:36 | |
you keep feeding that back into the process, and you can basically sample large sequences. | |
7:42 | |
By the end, you see that the transformer has learned about words and where to put spaces and where to put commas and so on. | |
7:48 | |
We're making more and more consistent predictions over time. These are the plots that you are looking at when you're doing model pretraining. | |
7:54 | |
Effectively, we're looking at the loss function over time as you train, and low loss means that our transformer | |
8:00 | |
is giving a higher probability to the next correct integer in the sequence. | |
8:06 | |
What are we going to do with model once we've trained it after a month? Well, the first thing that we noticed, we the field, | |
Base models learn powerful, general representations | |
8:14 | |
is that these models basically in the process of language modeling, learn very powerful general representations, | |
8:21 | |
and it's possible to very efficiently fine tune them for any arbitrary downstream tasks you might be interested in. | |
8:26 | |
As an example, if you're interested in sentiment classification, the approach used to be that you collect a bunch of positives | |
8:33 | |
and negatives and then you train some NLP model for that, but the new approach is: | |
8:38 | |
ignore sentiment classification, go off and do large language model pretraining, | |
8:43 | |
train a large transformer, and then you may only have a few examples and you can very efficiently fine tune | |
8:48 | |
your model for that task. This works very well in practice. The reason for this is that basically | |
8:55 | |
the transformer is forced to multitask a huge amount of tasks in the language modeling task, | |
9:00 | |
because in terms of predicting the next token, it's forced to understand a lot about the structure of the text and all the different concepts therein. | |
9:09 | |
That was GPT-1. Now around the time of GPT-2, people noticed that actually even better than fine tuning, | |
9:15 | |
you can actually prompt these models very effectively. These are language models and they want to complete documents, | |
9:20 | |
you can actually trick them into performing tasks by arranging these fake documents. | |
9:25 | |
In this example, for example, we have some passage and then we like do QA, QA, QA. | |
9:31 | |
This is called Few-shot prompt, and then we do Q, and then as the transformer is tried to complete the document is actually answering our question. | |
9:37 | |
This is an example of prompt engineering based model, making it believe that it's imitating a document and getting it to perform a task. | |
9:45 | |
This kicked off, I think the era of, I would say, prompting over fine tuning and seeing that this | |
9:50 | |
actually can work extremely well on a lot of problems, even without training any neural networks, fine tuning or so on. | |
9:56 | |
Now since then, we've seen an entire evolutionary tree of base models that everyone has trained. | |
10:02 | |
Not all of these models are available. for example, the GPT-4 base model was never released. | |
10:08 | |
The GPT-4 model that you might be interacting with over API is not a base model, it's an assistant model, and we're going to cover how to get those in a bit. | |
10:15 | |
GPT-3 based model is available via the API under the name Devanshi and GPT-2 based model | |
10:21 | |
is available even as weights on our GitHub repo. But currently the best available base model | |
10:27 | |
probably is the LLaMA series from Meta, although it is not commercially licensed. | |
10:32 | |
Now, one thing to point out is base models are not assistants. They don't want to make answers to your questions, | |
10:41 | |
they want to complete documents. If you tell them to write a poem about the bread and cheese, | |
10:46 | |
it will answer questions with more questions, it's completing what it thinks is a document. | |
10:51 | |
However, you can prompt them in a specific way for base models that is more likely to work. | |
10:57 | |
As an example, here's a poem about bread and cheese, and in that case it will autocomplete correctly. You can even trick base models into being assistants. | |
11:06 | |
The way you would do this is you would create a specific few-shot prompt that makes it look like there's some document between the human and assistant | |
11:13 | |
and they're exchanging information. Then at the bottom, you put your query at the end and the base model | |
11:21 | |
will condition itself into being a helpful assistant and answer, | |
11:26 | |
but this is not very reliable and doesn't work super well in practice, although it can be done. Instead, we have a different path to make | |
11:32 | |
actual GPT assistants not base model document completers. That takes us into supervised finetuning. | |
11:39 | |
In the supervised finetuning stage, we are going to collect small but high quality data-sets, and in this case, | |
11:45 | |
we're going to ask human contractors to gather data of the form prompt and ideal response. | |
11:52 | |
We're going to collect lots of these typically tens of thousands or something like that. Then we're going to still do language | |
11:58 | |
modeling on this data. Nothing changed algorithmically, we're swapping out a training set. It used to be Internet documents, | |
12:04 | |
which has a high quantity local for basically Q8 prompt response data. | |
12:11 | |
That is low quantity, high quality. We will still do language modeling and then after training, | |
12:16 | |
we get an SFT model. You can actually deploy these models and they are actual assistants and they work to some extent. | |
12:22 | |
Let me show you what an example demonstration might look like. Here's something that a human contractor might come up with. | |
12:28 | |
Here's some random prompt. Can you write a short introduction about the relevance of the term monopsony or something like that? | |
12:34 | |
Then the contractor also writes out an ideal response. When they write out these responses, they are following extensive labeling | |
12:40 | |
documentations and they are being asked to be helpful, truthful, and harmless. | |
12:45 | |
These labeling instructions here, you probably can't read it, neither can I, but they're long and this is people | |
12:52 | |
following instructions and trying to complete these prompts. That's what the dataset looks like. You can train these models. This works to some extent. | |
12:59 | |
Now, you can actually continue the pipeline from here on, and go into RLHF, | |
13:05 | |
reinforcement learning from human feedback that consists of both reward modeling and reinforcement learning. | |
13:10 | |
Let me cover that and then I'll come back to why you may want to go through the extra steps and how that compares to SFT models. | |
13:16 | |
In the reward modeling step, what we're going to do is we're now going to shift our data collection to be of the form of comparisons. | |
13:23 | |
Here's an example of what our dataset will look like. I have the same identical prompt on the top, | |
RM Dataset | |
13:28 | |
which is asking the assistant to write a program or a function that checks if a given string is a palindrome. | |
13:35 | |
Then what we do is we take the SFT model which we've already trained and we create multiple completions. | |
13:41 | |
In this case, we have three completions that the model has created, and then we ask people to rank these completions. | |
13:47 | |
If you stare at this for a while, and by the way, these are very difficult things to do to compare some of these predictions. | |
13:52 | |
This can take people even hours for a single prompt completion pairs, | |
13:57 | |
but let's say we decided that one of these is much better than the others and so on. We rank them. | |
14:03 | |
Then we can follow that with something that looks very much like a binary classification on all the possible pairs between these completions. | |
RM Training | |
14:10 | |
What we do now is, we lay out our prompt in rows, and the prompt is identical across all three rows here. | |
14:16 | |
It's all the same prompt, but the completion of this varies. The yellow tokens are coming from the SFT model. | |
14:21 | |
Then what we do is we append another special reward readout token at the end and we basically only | |
14:28 | |
supervise the transformer at this single green token. The transformer will predict some reward | |
14:34 | |
for how good that completion is for that prompt and basically it makes | |
14:39 | |
a guess about the quality of each completion. Then once it makes a guess for every one of them, | |
14:44 | |
we also have the ground truth which is telling us the ranking of them. We can actually enforce that some of | |
14:50 | |
these numbers should be much higher than others, and so on. We formulate this into a loss function and we train our model to make reward predictions | |
14:56 | |
that are consistent with the ground truth coming from the comparisons from all these contractors. That's how we train our reward model. | |
15:02 | |
That allows us to score how good a completion is for a prompt. Once we have a reward model, | |
15:09 | |
we can't deploy this because this is not very useful as an assistant by itself, but it's very useful for the reinforcement | |
15:15 | |
learning stage that follows now. Because we have a reward model, we can score the quality of any arbitrary completion for any given prompt. | |
15:22 | |
What we do during reinforcement learning is we basically get, again, a large collection of prompts and now we do | |
15:28 | |
reinforcement learning with respect to the reward model. Here's what that looks like. We take a single prompt, | |
15:34 | |
we lay it out in rows, and now we use basically the model we'd like to train which | |
15:39 | |
was initialized at SFT model to create some completions in yellow, and then we append the reward token again | |
15:45 | |
and we read off the reward according to the reward model, which is now kept fixed. It doesn't change any more. Now the reward model | |
15:53 | |
tells us the quality of every single completion for all these prompts and so what we can do is we can now just basically apply the same | |
15:59 | |
language modeling loss function, but we're currently training on the yellow tokens, and we are weighing | |
16:06 | |
the language modeling objective by the rewards indicated by the reward model. As an example, in the first row, | |
16:13 | |
the reward model said that this is a fairly high-scoring completion and so all the tokens that we | |
16:18 | |
happen to sample on the first row are going to get reinforced and they're going to get higher probabilities for the future. | |
16:25 | |
Conversely, on the second row, the reward model really did not like this completion, -1.2. Therefore, every single token that we sampled in | |
16:32 | |
that second row is going to get a slightly higher probability for the future. We do this over and over on many prompts on many batches and basically, | |
16:39 | |
we get a policy that creates yellow tokens here. It's basically all the completions here will | |
16:46 | |
score high according to the reward model that we trained in the previous stage. | |
16:51 | |
That's what the RLHF pipeline is. Then at the end, you get a model that you could deploy. | |
16:58 | |
As an example, ChatGPT is an RLHF model, but some other models that you might come across for example, | |
17:05 | |
Vicuna-13B, and so on, these are SFT models. We have base models, SFT models, and RLHF models. | |
17:12 | |
That's the state of things there. Now why would you want to do RLHF? One answer that's not | |
17:19 | |
that exciting is that it works better. This comes from the instruct GPT paper. According to these experiments a while ago now, | |
17:25 | |
these PPO models are RLHF. We see that they are basically preferred in a lot | |
17:30 | |
of comparisons when we give them to humans. Humans prefer basically tokens | |
17:36 | |
that come from RLHF models compared to SFT models, compared to base model that is prompted to be an assistant. It just works better. | |
17:43 | |
But you might ask why does it work better? I don't think that there's a single amazing answer | |
17:49 | |
that the community has really agreed on, but I will offer one reason potentially. | |
17:55 | |
It has to do with the asymmetry between how easy computationally it is to compare versus generate. | |
18:02 | |
Let's take an example of generating a haiku. Suppose I ask a model to write a haiku about paper clips. | |
18:07 | |
If you're a contractor trying to train data, then imagine being a contractor collecting basically data for the SFT stage, | |
18:14 | |
how are you supposed to create a nice haiku for a paper clip? You might not be very good at that, but if I give you a few examples of | |
18:20 | |
haikus you might be able to appreciate some of these haikus a lot more than others. Judging which one of these is good is a much easier task. | |
18:27 | |
Basically, this asymmetry makes it so that comparisons are a better way to potentially leverage | |
18:33 | |
yourself as a human and your judgment to create a slightly better model. Now, RLHF models are not | |
18:40 | |
strictly an improvement on the base models in some cases. In particular, we'd notice for example that they lose some entropy. | |
18:46 | |
That means that they give more peaky results. They can output samples | |
Mode collapse | |
18:54 | |
with lower variation than the base model. The base model has lots of entropy and will give lots of diverse outputs. | |
19:00 | |
For example, one place where I still prefer to use a base model is in the setup | |
19:06 | |
where you basically have n things and you want to generate more things like it. | |
19:13 | |
Here is an example that I just cooked up. I want to generate cool Pokemon names. | |
19:18 | |
I gave it seven Pokemon names and I asked the base model to complete the document and it gave me a lot more Pokemon names. | |
19:24 | |
These are fictitious. I tried to look them up. I don't believe they're actual Pokemons. This is the task that I think the base model would be | |
19:31 | |
good at because it still has lots of entropy. It'll give you lots of diverse cool more things that look like whatever you give it before. | |
19:41 | |
Having said all that, these are the assistant models that are probably available to you at this point. | |
19:47 | |
There was a team at Berkeley that ranked a lot of the available assistant models and give them basically Elo ratings. | |
19:53 | |
Currently, some of the best models, of course, are GPT-4, by far, I would say, followed by Claude, GPT-3.5, and then a number of models, | |
20:00 | |
some of these might be available as weights, like Vicuna, Koala, etc. The first three rows here are | |
20:07 | |
all RLHF models and all of the other models to my knowledge, are SFT models, I believe. | |
20:15 | |
That's how we train these models on the high level. Now I'm going to switch gears and let's look at how we can | |
20:22 | |
best apply the GPT assistant model to your problems. Now, I would like to work | |
20:27 | |
in setting of a concrete example. Let's work with a concrete example here. | |
20:32 | |
Let's say that you are working on an article or a blog post, and you're going to write this sentence at the end. | |
20:38 | |
"California's population is 53 times that of Alaska." So for some reason, you want to compare the populations of these two states. | |
20:44 | |
Think about the rich internal monologue and tool use and how much work actually goes computationally in | |
20:50 | |
your brain to generate this one final sentence. Here's maybe what that could look like in your brain. | |
20:55 | |
For this next step, let me blog on my blog, let me compare these two populations. | |
21:01 | |
First I'm going to obviously need to get both of these populations. Now, I know that I probably | |
21:06 | |
don't know these populations off the top of my head so I'm aware of what I know or don't know of my self-knowledge. | |
21:12 | |
I go, I do some tool use and I go to Wikipedia and I look up California's population and Alaska's population. | |
21:19 | |
Now, I know that I should divide the two, but again, I know that dividing 39.2 by 0.74 is very unlikely to succeed. | |
21:26 | |
That's not the thing that I can do in my head and so therefore, I'm going to rely on the calculator so I'm going to use a calculator, | |
21:33 | |
punch it in and see that the output is roughly 53. Then maybe I do some reflection and sanity checks in | |
21:40 | |
my brain so does 53 makes sense? Well, that's quite a large fraction, but then California is the most | |
21:45 | |
populous state, so maybe that looks okay. Then I have all the information I might need, and now I get to the creative portion of writing. | |
21:52 | |
I might start to write something like "California has 53x times greater" and then I think to myself, | |
21:58 | |
that's actually like really awkward phrasing so let me actually delete that and let me try again. | |
22:03 | |
As I'm writing, I have this separate process, almost inspecting what I'm writing and judging whether it looks good | |
22:09 | |
or not and then maybe I delete and maybe I reframe it, and then maybe I'm happy with what comes out. | |
22:15 | |
Basically long story short, a ton happens under the hood in terms of your internal monologue when you create sentences like this. | |
22:21 | |
But what does a sentence like this look like when we are training a GPT on it? From GPT's perspective, this | |
22:28 | |
is just a sequence of tokens. GPT, when it's reading or generating these tokens, | |
22:34 | |
it just goes chunk, chunk, chunk, chunk and each chunk is roughly the same amount of computational work for each token. | |
22:40 | |
These transformers are not very shallow networks they have about 80 layers of reasoning, | |
22:45 | |
but 80 is still not like too much. This transformer is going to do its best to imitate, | |
22:51 | |
but of course, the process here looks very different from the process that you took. In particular, in our final artifacts | |
22:59 | |
in the data sets that we create, and then eventually feed to LLMs, all that internal dialogue was completely stripped and unlike you, | |
23:07 | |
the GPT will look at every single token and spend the same amount of compute on every one of them. So, you can't expect it | |
23:13 | |
to do too much work per token and also in particular, | |
23:21 | |
basically these transformers are just like token simulators, they don't know what they don't know. | |
23:26 | |
They just imitate the next token. They don't know what they're good at or not good at. They just tried their best to imitate the next token. | |
23:32 | |
They don't reflect in the loop. They don't sanity check anything. They don't correct their mistakes along the way. | |
23:37 | |
By default, they just are sample token sequences. They don't have separate inner monologue streams | |
23:43 | |
in their head right? They're evaluating what's happening. Now, they do have some cognitive advantages, | |
23:48 | |
I would say and that is that they do actually have a very large fact-based knowledge across a vast number of areas because they have, | |
23:55 | |
say, several, 10 billion parameters. That's a lot of storage for a lot of facts. They also, I think have | |
24:02 | |
a relatively large and perfect working memory. Whatever fits into the context window | |
24:07 | |
is immediately available to the transformer through its internal self attention mechanism and so it's perfect memory, | |
24:14 | |
but it's got a finite size, but the transformer has a very direct access to it and so it can a losslessly remember anything that | |
24:22 | |
is inside its context window. This is how I would compare those two and the reason I bring all of this up is because I | |
24:27 | |
think to a large extent, prompting is just making up for this cognitive difference between | |
24:34 | |
these two architectures like our brains here and LLM brains. | |
24:39 | |
You can look at it that way almost. Here's one thing that people found for example works pretty well in practice. | |
24:45 | |
Especially if your tasks require reasoning, you can't expect the transformer to do too much reasoning per token. | |
24:52 | |
You have to really spread out the reasoning across more and more tokens. For example, you can't give a transformer | |
24:57 | |
a very complicated question and expect it to get the answer in a single token. There's just not enough time for it. "These transformers need tokens to | |
25:04 | |
think," I like to say sometimes. This is some of the things that work well, you may for example have a few-shot prompt that | |
25:10 | |
shows the transformer that it should show its work when it's answering question and if you give a few examples, | |
25:17 | |
the transformer will imitate that template and it will just end up working out better in terms of its evaluation. | |
25:24 | |
Additionally, you can elicit this behavior from the transformer by saying, let things step-by-step. | |
25:29 | |
Because this conditions the transformer into showing its work and because | |
25:34 | |
it snaps into a mode of showing its work, is going to do less computational work per token. | |
25:40 | |
It's more likely to succeed as a result because it's making slower reasoning over time. | |
25:46 | |
Here's another example, this one is called self-consistency. We saw that we had the ability | |
Ensemble multiple attempts | |
25:51 | |
to start writing and then if it didn't work out, I can try again and I can try multiple times | |
25:56 | |
and maybe select the one that worked best. In these approaches, | |
26:02 | |
you may sample not just once, but you may sample multiple times and then have some process for finding | |
26:07 | |
the ones that are good and then keeping just those samples or doing a majority vote or something like that. Basically these transformers in the process as | |
26:14 | |
they predict the next token, just like you, they can get unlucky and they could sample a not a very good | |
26:19 | |
token and they can go down like a blind alley in terms of reasoning. Unlike you, they cannot recover from that. | |
26:27 | |
They are stuck with every single token they sample and so they will continue the sequence, even if they know that this sequence is not going to work out. | |
26:34 | |
Give them the ability to look back, inspect or try to basically sample around it. | |
26:40 | |
Here's one technique also, it turns out that actually LLMs, they know when they've screwed up, | |
Ask for reflection | |
26:47 | |
so as an example, say you ask the model to generate a poem that does not | |
26:52 | |
rhyme and it might give you a poem, but it actually rhymes. But it turns out that especially for the bigger models like GPT-4, | |
26:58 | |
you can just ask it "did you meet the assignment?" Actually GPT-4 knows very well that it did not meet the assignment. | |
27:04 | |
It just got unlucky in its sampling. It will tell you, "No, I didn't actually meet the assignment here. Let me try again." | |
27:10 | |
But without you prompting it it doesn't know to revisit and so on. | |
27:17 | |
You have to make up for that in your prompts, and you have to get it to check, if you don't ask it to check, | |
27:23 | |
its not going to check by itself it's just a token simulator. | |
27:28 | |
I think more generally, a lot of these techniques fall into the bucket of what I would say recreating our System 2. | |
27:34 | |
You might be familiar with the System 1 and System 2 thinking for humans. System 1 is a fast automatic process and I | |
27:40 | |
think corresponds to an LLM just sampling tokens. System 2 is the slower deliberate | |
27:46 | |
planning part of your brain. This is a paper actually from | |
27:51 | |
just last week because this space is pretty quickly evolving, it's called Tree of Thought. | |
27:56 | |
The authors of this paper proposed maintaining multiple completions for any given prompt | |
28:02 | |
and then they are also scoring them along the way and keeping the ones that are going well if that makes sense. | |
28:08 | |
A lot of people are really playing around with prompt engineering | |
28:13 | |
to basically bring back some of these abilities that we have in our brain for LLMs. | |
28:19 | |
Now, one thing I would like to note here is that this is not just a prompt. This is actually prompts that are together | |
28:25 | |
used with some Python Glue code because you actually have to maintain multiple prompts and you also have to do | |
28:30 | |
some tree search algorithm here to figure out which prompts to expand, etc. It's a symbiosis of Python Glue code and | |
28:38 | |
individual prompts that are called in a while loop or in a bigger algorithm. I also think there's a really cool | |
28:43 | |
parallel here to AlphaGo. AlphaGo has a policy for placing the next stone when it plays go, | |
28:48 | |
and its policy was trained originally by imitating humans. But in addition to this policy, | |
28:54 | |
it also does Monte Carlo Tree Search. Basically, it will play out a number of possibilities in its head and evaluate all of | |
29:00 | |
them and only keep the ones that work well. I think this is an equivalent of AlphaGo but for text if that makes sense. | |
29:08 | |
Just like Tree of Thought, I think more generally people are starting to really explore | |
29:13 | |
more general techniques of not just the simple question-answer prompts, but something that looks a lot more like | |
29:19 | |
Python Glue code stringing together many prompts. On the right, I have an example from this paper called React where they | |
29:25 | |
structure the answer to a prompt as a sequence of thought-action-observation, | |
29:32 | |
thought-action-observation, and it's a full rollout and a thinking process to answer the query. | |
29:38 | |
In these actions, the model is also allowed to tool use. On the left, I have an example of AutoGPT. | |
29:45 | |
Now AutoGPT by the way is a project that I think got a lot of hype recently, | |
29:51 | |
but I think I still find it inspirationally interesting. It's a project that allows an LLM to keep | |
29:58 | |
the task list and continue to recursively break down tasks. I don't think this currently works very well and I would | |
30:04 | |
not advise people to use it in practical applications. I just think it's something to generally take inspiration | |
30:09 | |
from in terms of where this is going, I think over time. That's like giving our model System 2 thinking. | |
30:16 | |
The next thing I find interesting is, this following serve I would say almost psychological quirk of LLMs, | |
30:23 | |
is that LLMs don't want to succeed, they want to imitate. You want to succeed, and you should ask for it. | |
30:31 | |
What I mean by that is, when transformers are trained, they have training sets and there can be | |
30:38 | |
an entire spectrum of performance qualities in their training data. For example, there could be some kind of a prompt | |
30:43 | |
for some physics question or something like that, and there could be a student's solution that is completely wrong but there can also be an expert | |
30:49 | |
answer that is extremely right. Transformers can't tell the difference between low, | |
30:54 | |
they know about low-quality solutions and high-quality solutions, but by default, they want to imitate all of | |
30:59 | |
it because they're just trained on language modeling. At test time, you actually have to ask for a good performance. | |
31:06 | |
In this example in this paper, they tried various prompts. Let's think step-by-step was very powerful | |
31:13 | |
because it spread out the reasoning over many tokens. But what worked even better is, let's work this out in a step-by-step way | |
31:19 | |
to be sure we have the right answer. It's like conditioning on getting the right answer, and this actually makes the transformer work | |
31:25 | |
better because the transformer doesn't have to now hedge its probability mass on low-quality solutions, | |
31:31 | |
as ridiculous as that sounds. Basically, feel free to ask for a strong solution. | |
31:37 | |
Say something like, you are a leading expert on this topic. Pretend you have IQ 120, etc. But don't try to ask for too much IQ because if | |
31:44 | |
you ask for IQ 400, you might be out of data distribution, or even worse, you could be in data distribution for | |
31:51 | |
something like sci-fi stuff and it will start to take on some sci-fi, or like roleplaying or something like that. | |
31:56 | |
You have to find the right amount of IQ. I think it's got some U-shaped curve there. | |
32:02 | |
Next up, as we saw when we are trying to solve problems, we know what we are good at and what we're not good at, | |
32:09 | |
and we lean on tools computationally. You want to do the same potentially with your LLMs. | |
Tool use / Plugins | |
32:15 | |
In particular, we may want to give them calculators, code interpreters, | |
32:20 | |
and so on, the ability to do search, and there's a lot of techniques for doing that. | |
32:27 | |
One thing to keep in mind, again, is that these transformers by default may not know what they don't know. | |
32:32 | |
You may even want to tell the transformer in a prompt you are not very good at mental arithmetic. Whenever you need to do very large number addition, | |
32:40 | |
multiplication, or whatever, instead, use this calculator. Here's how you use the calculator, you use this token combination, etc. | |
32:46 | |
You have to actually spell it out because the model by default doesn't know what it's good at or not good at, necessarily, just like you and I might be. | |
32:54 | |
Next up, I think something that is very interesting is we went from a world that was retrieval only all the way, | |
33:02 | |
the pendulum has swung to the other extreme where its memory only in LLMs. But actually, there's this entire space in-between of | |
33:08 | |
these retrieval-augmented models and this works extremely well in practice. As I mentioned, the context window of | |
33:14 | |
a transformer is its working memory. If you can load the working memory with any information that is relevant to the task, | |
33:21 | |
the model will work extremely well because it can immediately access all that memory. I think a lot of people are really interested | |
33:28 | |
in basically retrieval-augment degeneration. On the bottom, I have an example of LlamaIndex which is | |
33:35 | |
one data connector to lots of different types of data. You can index all | |
33:41 | |
of that data and you can make it accessible to LLMs. The emerging recipe there is you take relevant documents, | |
33:47 | |
you split them up into chunks, you embed all of them, and you basically get embedding vectors that represent that data. | |
33:53 | |
You store that in the vector store and then at test time, you make some kind of a query to your vector store and you fetch chunks that | |
34:00 | |
might be relevant to your task and you stuff them into the prompt and then you generate. This can work quite well in practice. | |
34:06 | |
This is, I think, similar to when you and I solve problems. You can do everything from your memory and | |
34:11 | |
transformers have very large and extensive memory, but also it really helps to reference some primary documents. | |
34:17 | |
Whenever you find yourself going back to a textbook to find something, or whenever you find yourself going back to documentation of the library to look something up, | |
34:25 | |
transformers definitely want to do that too. You have some memory over how | |
34:30 | |
some documentation of the library works but it's much better to look it up. The same applies here. | |
34:35 | |
Next, I wanted to briefly talk about constraint prompting. I also find this very interesting. | |
34:41 | |
This is basically techniques for forcing a certain template in the outputs of LLMs. | |
34:50 | |
Guidance is one example from Microsoft actually. Here we are enforcing that the output from the LLM will be JSON. | |
34:57 | |
This will actually guarantee that the output will take on this form because they go in and they mess with the probabilities of | |
35:03 | |
all the different tokens that come out of the transformer and they clamp those tokens and then the transformer is only filling in the blanks here, | |
35:09 | |
and then you can enforce additional restrictions on what could go into those blanks. This might be really helpful, and I think | |
35:15 | |
this constraint sampling is also extremely interesting. I also want to say | |
35:20 | |
a few words about fine tuning. It is the case that you can get really far with prompt engineering, but it's also possible to | |
35:27 | |
think about fine tuning your models. Now, fine tuning models means that you are actually going to change the weights of the model. | |
35:33 | |
It is becoming a lot more accessible to do this in practice, and that's because of a number of techniques that have been | |
35:39 | |
developed and have libraries for very recently. So for example parameter efficient fine tuning techniques like Laura, | |
35:46 | |
make sure that you're only training small, sparse pieces of your model. So most of the model is kept clamped at | |
35:53 | |
the base model and some pieces of it are allowed to change and this still works pretty well empirically and makes | |
35:58 | |
it much cheaper to tune only small pieces of your model. It also means that because most of your model is clamped, | |
36:05 | |
you can use very low precision inference for computing those parts because you are not going to be updated by | |
36:10 | |
gradient descent and so that makes everything a lot more efficient as well. And in addition, we have a number of open source, high-quality base models. | |
36:17 | |
Currently, as I mentioned, I think LLaMa is quite nice, although it is not commercially licensed, I believe right now. | |
36:23 | |
Some things to keep in mind is that basically fine tuning is a lot more technically involved. | |
36:29 | |
It requires a lot more, I think, technical expertise to do right. It requires human data contractors for | |
36:34 | |
datasets and/or synthetic data pipelines that can be pretty complicated. This will definitely slow down | |
36:40 | |
your iteration cycle by a lot, and I would say on a high level SFT is achievable because you're continuing | |
36:47 | |
the language modeling task. It's relatively straightforward, but RLHF, I would say is very much research territory | |
36:53 | |
and is even much harder to get to work, and so I would probably not advise that someone just tries to roll their own RLHF of implementation. | |
37:00 | |
These things are pretty unstable, very difficult to train, not something that is, I think, very beginner friendly right now, | |
37:06 | |
and it's also potentially likely also to change pretty rapidly still. | |
37:11 | |
So I think these are my default recommendations right now. I would break up your task into two major parts. | |
Default recommendations | |
37:18 | |
Number 1, achieve your top performance, and Number 2, optimize your performance in that order. | |
37:23 | |
Number 1, the best performance will currently come from GPT-4 model. It is the most capable of all by far. | |
37:29 | |
Use prompts that are very detailed. They have lots of task content, relevant information and instructions. | |
37:36 | |
Think along the lines of what would you tell a task contractor if they can't email you back, but then also keep in mind that a task contractor is a | |
37:43 | |
human and they have inner monologue and they're very clever, etc. LLMs do not possess those qualities. | |
37:48 | |
So make sure to think through the psychology of the LLM almost and cater prompts to that. | |
37:54 | |
Retrieve and add any relevant context and information to these prompts. Basically refer to a lot of | |
38:01 | |
the prompt engineering techniques. Some of them I've highlighted in the slides above, but also this is a very large space and I would | |
38:07 | |
just advise you to look for prompt engineering techniques online. There's a lot to cover there. | |
38:13 | |
Experiment with few-shot examples. What this refers to is, you don't just want to tell, you want to show whenever it's possible. | |
38:19 | |
So give it examples of everything that helps it really understand what you mean if you can. | |
38:25 | |
Experiment with tools and plug-ins to offload tasks that are difficult for LLMs natively, | |
38:30 | |
and then think about not just a single prompt and answer, think about potential chains and reflection and how you glue | |
38:36 | |
them together and how you can potentially make multiple samples and so on. Finally, if you think you've squeezed | |
38:42 | |
out prompt engineering, which I think you should stick with for a while, look at some potentially | |
38:48 | |
fine tuning a model to your application, but expect this to be a lot more slower in the vault and then | |
38:54 | |
there's an expert fragile research zone here and I would say that is RLHF, which currently does work a bit | |
39:00 | |
better than SFT if you can get it to work. But again, this is pretty involved, I would say. And to optimize your costs, | |
39:06 | |
try to explore lower capacity models or shorter prompts and so on. | |
39:12 | |
I also wanted to say a few words about the use cases in which I think LLMs are currently well suited for. | |
39:18 | |
In particular, note that there's a large number of limitations to LLMs today, and so I would keep that | |
39:24 | |
definitely in mind for all of your applications. Models, and this by the way could be an entire talk. So I don't have time to cover it in full detail. | |
39:30 | |
Models may be biased, they may fabricate, hallucinate information, they may have reasoning errors, they may struggle in entire classes of applications, | |
39:38 | |
they have knowledge cut-offs, so they might not know any information above, say, September, 2021. | |
39:43 | |
They are susceptible to a large range of attacks which are coming out on Twitter daily, | |
39:48 | |
including prompt injection, jailbreak attacks, data poisoning attacks and so on. So my recommendation right now is | |
39:54 | |
use LLMs in low-stakes applications. Combine them always with human oversight. | |
40:00 | |
Use them as a source of inspiration and suggestions and think co-pilots, instead of completely autonomous agents | |
40:05 | |
that are just like performing a task somewhere. It's just not clear that the models are there right now. | |
40:11 | |
So I wanted to close by saying that GPT-4 is an amazing artifact. I'm very thankful that it exists, and it's beautiful. | |
40:18 | |
It has a ton of knowledge across so many areas. It can do math, code and so on. And in addition, there's this | |
40:24 | |
thriving ecosystem of everything else that is being built and incorporated into the ecosystem. Some of these things I've talked about, | |
40:31 | |
and all of this power is accessible at your fingertips. So here's everything that's needed in terms of | |
40:37 | |
code to ask GPT-4 a question, to prompt it, and get a response. In this case, I said, | |
40:44 | |
can you say something to inspire the audience of Microsoft Build 2023? And I just punched this into Python and verbatim | |
40:50 | |
GPT-4 said the following: And by the way, I did not know that they | |
40:55 | |
used this trick in the keynote. So I thought I was being clever, but it is really good at this. | |
41:02 | |
It says, ladies and gentlemen, innovators and trailblazers Microsoft Build 2023. Welcome to the gathering of brilliant | |
41:08 | |
minds like no other, you are the architects of the future, the visionaries molding the digital realm | |
41:13 | |
in which humanity thrives. Embrace the limitless possibilities of technologies and let your ideas soar as high as your imagination. | |
41:20 | |
Together, let's create a more connected, remarkable, and inclusive world for generations to come. Get ready to unleash your creativity, | |
41:27 | |
canvas the unknown, and turn dreams into reality. Your journey begins today! | |