Upload 2 files

State of GPT - Andrej Karpathy.txt

@@ -0,0 +1,812 @@
+0:00
+[MUSIC]
+0:07
+ANNOUNCER: Please welcome AI researcher and founding member of OpenAI, Andrej Karpathy.
+0:21
+ANDREJ KARPATHY: Hi, everyone. I'm happy to be here to tell you about the state of GPT and more generally about
+0:28
+the rapidly growing ecosystem of large language models. I would like to partition the talk into two parts.
+0:35
+In the first part, I would like to tell you about how we train GPT Assistance, and then in the second part,
+0:40
+we're going to take a look at how we can use these assistants effectively for your applications.
+0:46
+First, let's take a look at the emerging recipe for how to train these assistants and keep in mind that this is all very new and still rapidly evolving,
+0:53
+but so far, the recipe looks something like this. Now, this is a complicated slide, I'm going to go through it piece by
+GPT Assistant training pipeline
+0:59
+piece, but roughly speaking, we have four major stages, pretraining,
+1:04
+supervised finetuning, reward modeling, reinforcement learning, and they follow each other serially.
+1:09
+Now, in each stage, we have a dataset that powers that stage. We have an algorithm that for our purposes will be
+1:17
+a objective and over for training the neural network, and then we have a resulting model,
+1:23
+and then there are some notes on the bottom. The first stage we're going to start with as the pretraining stage. Now, this stage is special in this diagram,
+1:31
+and this diagram is not to scale because this stage is where all of the computational work basically happens. This is 99 percent of the training
+1:38
+compute time and also flops. This is where we are dealing with
+1:44
+Internet scale datasets with thousands of GPUs in the supercomputer and also months of training potentially.
+1:51
+The other three stages are finetuning stages that are much more along the lines of small few number of GPUs and hours or days.
+1:59
+Let's take a look at the pretraining stage to achieve a base model. First, we are going to gather a large amount of data.
+Data collection
+2:07
+Here's an example of what we call a data mixture that comes from this paper that was released by
+2:13
+Meta where they released this LLaMA based model. Now, you can see roughly the datasets that
+2:18
+enter into these collections. We have CommonCrawl, which is a web scrape, C4, which is also CommonCrawl,
+2:25
+and then some high quality datasets as well. For example, GitHub, Wikipedia, Books, Archives, Stock Exchange and so on.
+2:31
+These are all mixed up together, and then they are sampled according to some given proportions,
+2:36
+and that forms the training set for the GPT. Now before we can actually train on this data,
+2:43
+we need to go through one more preprocessing step, and that is tokenization. This is basically a translation of
+2:48
+the raw text that we scrape from the Internet into sequences of integers because
+2:53
+that's the native representation over which GPTs function. Now, this is a lossless translation
+3:00
+between pieces of texts and tokens and integers, and there are a number of algorithms for the stage.
+3:05
+Typically, for example, you could use something like byte pair encoding, which iteratively merges text chunks
+3:11
+and groups them into tokens. Here, I'm showing some example chunks of these tokens,
+3:16
+and then this is the raw integer sequence that will actually feed into a transformer. Now, here I'm showing
+2 example models
+3:23
+two examples for hybrid parameters that govern this stage.
+3:28
+GPT-4, we did not release too much information about how it was trained and so on, I'm using GPT-3s numbers,
+3:33
+but GPT-3 is of course a little bit old by now, about three years ago. But LLaMA is a fairly recent model from Meta.
+3:40
+These are roughly the orders of magnitude that we're dealing with when we're doing pretraining. The vocabulary size is usually a couple 10,000 tokens.
+3:48
+The context length is usually something like 2,000, 4,000, or nowadays even 100,000,
+3:53
+and this governs the maximum number of integers that the GPT will look at when it's trying to
+3:58
+predict the next integer in a sequence. You can see that roughly the number of parameters say,
+4:04
+65 billion for LLaMA. Now, even though LLaMA has only 65B parameters compared to GPP-3s 175 billion parameters,
+4:11
+LLaMA is a significantly more powerful model, and intuitively, that's because the model is trained for significantly longer.
+4:17
+In this case, 1.4 trillion tokens, instead of 300 billion tokens. You shouldn't judge the power of a model by
+4:23
+the number of parameters that it contains. Below, I'm showing some tables of rough hyperparameters that typically
+4:31
+go into specifying the transformer neural network, the number of heads, the dimension size, number of layers,
+4:36
+and so on, and on the bottom I'm showing some training hyperparameters. For example, to train the 65B model,
+4:44
+Meta used 2,000 GPUs, roughly 21 days of training and a roughly several million dollars.
+4:52
+That's the rough orders of magnitude that you should have in mind for the pre-training stage.
+4:57
+Now, when we're actually pre-training, what happens? Roughly speaking, we are going to take our tokens,
+5:03
+and we're going to lay them out into data batches. We have these arrays that will feed into the transformer,
+5:09
+and these arrays are B, the batch size and these are all independent examples stocked up in rows and B by T,
+5:16
+T being the maximum context length. In my picture I only have 10 the context lengths, so this could be 2,000, 4,000, etc.
+5:23
+These are extremely long rows. What we do is we take these documents, and we pack them into rows,
+5:28
+and we delimit them with these special end of texts tokens, basically telling the transformer where a new document begins.
+5:35
+Here, I have a few examples of documents and then I stretch them out into this input.
+5:41
+Now, we're going to feed all of these numbers into transformer. Let me just focus on a single particular cell,
+5:49
+but the same thing will happen at every cell in this diagram. Let's look at the green cell. The green cell is going to take
+5:56
+a look at all of the tokens before it, so all of the tokens in yellow, and we're going to feed that entire context
+6:03
+into the transforming neural network, and the transformer is going to try to predict the next token in
+6:08
+a sequence, in this case in red. Now the transformer, I don't have too much time to, unfortunately, go into the full details of this
+6:14
+neural network architecture is just a large blob of neural net stuff for our purposes, and it's got several,
+6:20
+10 billion parameters typically or something like that. Of course, as I tune these parameters, you're getting slightly different predicted distributions
+6:26
+for every single one of these cells. For example, if our vocabulary size is 50,257 tokens,
+6:34
+then we're going to have that many numbers because we need to specify a probability distribution for what comes next.
+6:40
+Basically, we have a probability for whatever may follow. Now, in this specific example, for this specific cell,
+6:45
+513 will come next, and so we can use this as a source of supervision to update our transformers weights.
+6:51
+We're applying this basically on every single cell in the parallel, and we keep swapping batches, and we're trying to get the transformer to make
+6:58
+the correct predictions over what token comes next in a sequence. Let me show you more concretely what this looks
+7:03
+like when you train one of these models. This is actually coming from the New York Times, and they trained a small GPT on Shakespeare.
+7:11
+Here's a small snippet of Shakespeare, and they train their GPT on it. Now, in the beginning, at initialization,
+7:17
+the GPT starts with completely random weights. You're getting completely random outputs as well. But over time, as you train the GPT longer and longer,
+7:26
+you are getting more and more coherent and consistent samples from the model,
+7:31
+and the way you sample from it, of course, is you predict what comes next, you sample from that distribution and
+7:36
+you keep feeding that back into the process, and you can basically sample large sequences.
+7:42
+By the end, you see that the transformer has learned about words and where to put spaces and where to put commas and so on.
+7:48
+We're making more and more consistent predictions over time. These are the plots that you are looking at when you're doing model pretraining.
+7:54
+Effectively, we're looking at the loss function over time as you train, and low loss means that our transformer
+8:00
+is giving a higher probability to the next correct integer in the sequence.
+8:06
+What are we going to do with model once we've trained it after a month? Well, the first thing that we noticed, we the field,
+Base models learn powerful, general representations
+8:14
+is that these models basically in the process of language modeling, learn very powerful general representations,
+8:21
+and it's possible to very efficiently fine tune them for any arbitrary downstream tasks you might be interested in.
+8:26
+As an example, if you're interested in sentiment classification, the approach used to be that you collect a bunch of positives
+8:33
+and negatives and then you train some NLP model for that, but the new approach is:
+8:38
+ignore sentiment classification, go off and do large language model pretraining,
+8:43
+train a large transformer, and then you may only have a few examples and you can very efficiently fine tune
+8:48
+your model for that task. This works very well in practice. The reason for this is that basically
+8:55
+the transformer is forced to multitask a huge amount of tasks in the language modeling task,
+9:00
+because in terms of predicting the next token, it's forced to understand a lot about the structure of the text and all the different concepts therein.
+9:09
+That was GPT-1. Now around the time of GPT-2, people noticed that actually even better than fine tuning,
+9:15
+you can actually prompt these models very effectively. These are language models and they want to complete documents,
+9:20
+you can actually trick them into performing tasks by arranging these fake documents.
+9:25
+In this example, for example, we have some passage and then we like do QA, QA, QA.
+9:31
+This is called Few-shot prompt, and then we do Q, and then as the transformer is tried to complete the document is actually answering our question.
+9:37
+This is an example of prompt engineering based model, making it believe that it's imitating a document and getting it to perform a task.
+9:45
+This kicked off, I think the era of, I would say, prompting over fine tuning and seeing that this
+9:50
+actually can work extremely well on a lot of problems, even without training any neural networks, fine tuning or so on.
+9:56
+Now since then, we've seen an entire evolutionary tree of base models that everyone has trained.
+10:02
+Not all of these models are available. for example, the GPT-4 base model was never released.
+10:08
+The GPT-4 model that you might be interacting with over API is not a base model, it's an assistant model, and we're going to cover how to get those in a bit.
+10:15
+GPT-3 based model is available via the API under the name Devanshi and GPT-2 based model
+10:21
+is available even as weights on our GitHub repo. But currently the best available base model
+10:27
+probably is the LLaMA series from Meta, although it is not commercially licensed.
+10:32
+Now, one thing to point out is base models are not assistants. They don't want to make answers to your questions,
+10:41
+they want to complete documents. If you tell them to write a poem about the bread and cheese,
+10:46
+it will answer questions with more questions, it's completing what it thinks is a document.
+10:51
+However, you can prompt them in a specific way for base models that is more likely to work.
+10:57
+As an example, here's a poem about bread and cheese, and in that case it will autocomplete correctly. You can even trick base models into being assistants.
+11:06
+The way you would do this is you would create a specific few-shot prompt that makes it look like there's some document between the human and assistant
+11:13
+and they're exchanging information. Then at the bottom, you put your query at the end and the base model
+11:21
+will condition itself into being a helpful assistant and answer,
+11:26
+but this is not very reliable and doesn't work super well in practice, although it can be done. Instead, we have a different path to make
+11:32
+actual GPT assistants not base model document completers. That takes us into supervised finetuning.
+11:39
+In the supervised finetuning stage, we are going to collect small but high quality data-sets, and in this case,
+11:45
+we're going to ask human contractors to gather data of the form prompt and ideal response.
+11:52
+We're going to collect lots of these typically tens of thousands or something like that. Then we're going to still do language
+11:58
+modeling on this data. Nothing changed algorithmically, we're swapping out a training set. It used to be Internet documents,
+12:04
+which has a high quantity local for basically Q8 prompt response data.
+12:11
+That is low quantity, high quality. We will still do language modeling and then after training,
+12:16
+we get an SFT model. You can actually deploy these models and they are actual assistants and they work to some extent.
+12:22
+Let me show you what an example demonstration might look like. Here's something that a human contractor might come up with.
+12:28
+Here's some random prompt. Can you write a short introduction about the relevance of the term monopsony or something like that?
+12:34
+Then the contractor also writes out an ideal response. When they write out these responses, they are following extensive labeling
+12:40
+documentations and they are being asked to be helpful, truthful, and harmless.
+12:45
+These labeling instructions here, you probably can't read it, neither can I, but they're long and this is people
+12:52
+following instructions and trying to complete these prompts. That's what the dataset looks like. You can train these models. This works to some extent.
+12:59
+Now, you can actually continue the pipeline from here on, and go into RLHF,
+13:05
+reinforcement learning from human feedback that consists of both reward modeling and reinforcement learning.
+13:10
+Let me cover that and then I'll come back to why you may want to go through the extra steps and how that compares to SFT models.
+13:16
+In the reward modeling step, what we're going to do is we're now going to shift our data collection to be of the form of comparisons.
+13:23
+Here's an example of what our dataset will look like. I have the same identical prompt on the top,
+RM Dataset
+13:28
+which is asking the assistant to write a program or a function that checks if a given string is a palindrome.
+13:35
+Then what we do is we take the SFT model which we've already trained and we create multiple completions.
+13:41
+In this case, we have three completions that the model has created, and then we ask people to rank these completions.
+13:47
+If you stare at this for a while, and by the way, these are very difficult things to do to compare some of these predictions.
+13:52
+This can take people even hours for a single prompt completion pairs,
+13:57
+but let's say we decided that one of these is much better than the others and so on. We rank them.
+14:03
+Then we can follow that with something that looks very much like a binary classification on all the possible pairs between these completions.
+RM Training
+14:10
+What we do now is, we lay out our prompt in rows, and the prompt is identical across all three rows here.
+14:16
+It's all the same prompt, but the completion of this varies. The yellow tokens are coming from the SFT model.
+14:21
+Then what we do is we append another special reward readout token at the end and we basically only
+14:28
+supervise the transformer at this single green token. The transformer will predict some reward
+14:34
+for how good that completion is for that prompt and basically it makes
+14:39
+a guess about the quality of each completion. Then once it makes a guess for every one of them,
+14:44
+we also have the ground truth which is telling us the ranking of them. We can actually enforce that some of
+14:50
+these numbers should be much higher than others, and so on. We formulate this into a loss function and we train our model to make reward predictions
+14:56
+that are consistent with the ground truth coming from the comparisons from all these contractors. That's how we train our reward model.
+15:02
+That allows us to score how good a completion is for a prompt. Once we have a reward model,
+15:09
+we can't deploy this because this is not very useful as an assistant by itself, but it's very useful for the reinforcement
+15:15
+learning stage that follows now. Because we have a reward model, we can score the quality of any arbitrary completion for any given prompt.
+15:22
+What we do during reinforcement learning is we basically get, again, a large collection of prompts and now we do
+15:28
+reinforcement learning with respect to the reward model. Here's what that looks like. We take a single prompt,
+15:34
+we lay it out in rows, and now we use basically the model we'd like to train which
+15:39
+was initialized at SFT model to create some completions in yellow, and then we append the reward token again
+15:45
+and we read off the reward according to the reward model, which is now kept fixed. It doesn't change any more. Now the reward model
+15:53
+tells us the quality of every single completion for all these prompts and so what we can do is we can now just basically apply the same
+15:59
+language modeling loss function, but we're currently training on the yellow tokens, and we are weighing
+16:06
+the language modeling objective by the rewards indicated by the reward model. As an example, in the first row,
+16:13
+the reward model said that this is a fairly high-scoring completion and so all the tokens that we
+16:18
+happen to sample on the first row are going to get reinforced and they're going to get higher probabilities for the future.
+16:25
+Conversely, on the second row, the reward model really did not like this completion, -1.2. Therefore, every single token that we sampled in
+16:32
+that second row is going to get a slightly higher probability for the future. We do this over and over on many prompts on many batches and basically,
+16:39
+we get a policy that creates yellow tokens here. It's basically all the completions here will
+16:46
+score high according to the reward model that we trained in the previous stage.
+16:51
+That's what the RLHF pipeline is. Then at the end, you get a model that you could deploy.
+16:58
+As an example, ChatGPT is an RLHF model, but some other models that you might come across for example,
+17:05
+Vicuna-13B, and so on, these are SFT models. We have base models, SFT models, and RLHF models.
+17:12
+That's the state of things there. Now why would you want to do RLHF? One answer that's not
+17:19
+that exciting is that it works better. This comes from the instruct GPT paper. According to these experiments a while ago now,
+17:25
+these PPO models are RLHF. We see that they are basically preferred in a lot
+17:30
+of comparisons when we give them to humans. Humans prefer basically tokens
+17:36
+that come from RLHF models compared to SFT models, compared to base model that is prompted to be an assistant. It just works better.
+17:43
+But you might ask why does it work better? I don't think that there's a single amazing answer
+17:49
+that the community has really agreed on, but I will offer one reason potentially.
+17:55
+It has to do with the asymmetry between how easy computationally it is to compare versus generate.
+18:02
+Let's take an example of generating a haiku. Suppose I ask a model to write a haiku about paper clips.
+18:07
+If you're a contractor trying to train data, then imagine being a contractor collecting basically data for the SFT stage,
+18:14
+how are you supposed to create a nice haiku for a paper clip? You might not be very good at that, but if I give you a few examples of
+18:20
+haikus you might be able to appreciate some of these haikus a lot more than others. Judging which one of these is good is a much easier task.
+18:27
+Basically, this asymmetry makes it so that comparisons are a better way to potentially leverage
+18:33
+yourself as a human and your judgment to create a slightly better model. Now, RLHF models are not
+18:40
+strictly an improvement on the base models in some cases. In particular, we'd notice for example that they lose some entropy.
+18:46
+That means that they give more peaky results. They can output samples
+Mode collapse
+18:54
+with lower variation than the base model. The base model has lots of entropy and will give lots of diverse outputs.
+19:00
+For example, one place where I still prefer to use a base model is in the setup
+19:06
+where you basically have n things and you want to generate more things like it.
+19:13
+Here is an example that I just cooked up. I want to generate cool Pokemon names.
+19:18
+I gave it seven Pokemon names and I asked the base model to complete the document and it gave me a lot more Pokemon names.
+19:24
+These are fictitious. I tried to look them up. I don't believe they're actual Pokemons. This is the task that I think the base model would be
+19:31
+good at because it still has lots of entropy. It'll give you lots of diverse cool more things that look like whatever you give it before.
+19:41
+Having said all that, these are the assistant models that are probably available to you at this point.
+19:47
+There was a team at Berkeley that ranked a lot of the available assistant models and give them basically Elo ratings.
+19:53
+Currently, some of the best models, of course, are GPT-4, by far, I would say, followed by Claude, GPT-3.5, and then a number of models,
+20:00
+some of these might be available as weights, like Vicuna, Koala, etc. The first three rows here are
+20:07
+all RLHF models and all of the other models to my knowledge, are SFT models, I believe.
+20:15
+That's how we train these models on the high level. Now I'm going to switch gears and let's look at how we can
+20:22
+best apply the GPT assistant model to your problems. Now, I would like to work
+20:27
+in setting of a concrete example. Let's work with a concrete example here.
+20:32
+Let's say that you are working on an article or a blog post, and you're going to write this sentence at the end.
+20:38
+"California's population is 53 times that of Alaska." So for some reason, you want to compare the populations of these two states.
+20:44
+Think about the rich internal monologue and tool use and how much work actually goes computationally in
+20:50
+your brain to generate this one final sentence. Here's maybe what that could look like in your brain.
+20:55
+For this next step, let me blog on my blog, let me compare these two populations.
+21:01
+First I'm going to obviously need to get both of these populations. Now, I know that I probably
+21:06
+don't know these populations off the top of my head so I'm aware of what I know or don't know of my self-knowledge.
+21:12
+I go, I do some tool use and I go to Wikipedia and I look up California's population and Alaska's population.
+21:19
+Now, I know that I should divide the two, but again, I know that dividing 39.2 by 0.74 is very unlikely to succeed.
+21:26
+That's not the thing that I can do in my head and so therefore, I'm going to rely on the calculator so I'm going to use a calculator,
+21:33
+punch it in and see that the output is roughly 53. Then maybe I do some reflection and sanity checks in
+21:40
+my brain so does 53 makes sense? Well, that's quite a large fraction, but then California is the most
+21:45
+populous state, so maybe that looks okay. Then I have all the information I might need, and now I get to the creative portion of writing.
+21:52
+I might start to write something like "California has 53x times greater" and then I think to myself,
+21:58
+that's actually like really awkward phrasing so let me actually delete that and let me try again.
+22:03
+As I'm writing, I have this separate process, almost inspecting what I'm writing and judging whether it looks good
+22:09
+or not and then maybe I delete and maybe I reframe it, and then maybe I'm happy with what comes out.
+22:15
+Basically long story short, a ton happens under the hood in terms of your internal monologue when you create sentences like this.
+22:21
+But what does a sentence like this look like when we are training a GPT on it? From GPT's perspective, this
+22:28
+is just a sequence of tokens. GPT, when it's reading or generating these tokens,
+22:34
+it just goes chunk, chunk, chunk, chunk and each chunk is roughly the same amount of computational work for each token.
+22:40
+These transformers are not very shallow networks they have about 80 layers of reasoning,
+22:45
+but 80 is still not like too much. This transformer is going to do its best to imitate,
+22:51
+but of course, the process here looks very different from the process that you took. In particular, in our final artifacts
+22:59
+in the data sets that we create, and then eventually feed to LLMs, all that internal dialogue was completely stripped and unlike you,
+23:07
+the GPT will look at every single token and spend the same amount of compute on every one of them. So, you can't expect it
+23:13
+to do too much work per token and also in particular,
+23:21
+basically these transformers are just like token simulators, they don't know what they don't know.
+23:26
+They just imitate the next token. They don't know what they're good at or not good at. They just tried their best to imitate the next token.
+23:32
+They don't reflect in the loop. They don't sanity check anything. They don't correct their mistakes along the way.
+23:37
+By default, they just are sample token sequences. They don't have separate inner monologue streams
+23:43
+in their head right? They're evaluating what's happening. Now, they do have some cognitive advantages,
+23:48
+I would say and that is that they do actually have a very large fact-based knowledge across a vast number of areas because they have,
+23:55
+say, several, 10 billion parameters. That's a lot of storage for a lot of facts. They also, I think have
+24:02
+a relatively large and perfect working memory. Whatever fits into the context window
+24:07
+is immediately available to the transformer through its internal self attention mechanism and so it's perfect memory,
+24:14
+but it's got a finite size, but the transformer has a very direct access to it and so it can a losslessly remember anything that
+24:22
+is inside its context window. This is how I would compare those two and the reason I bring all of this up is because I
+24:27
+think to a large extent, prompting is just making up for this cognitive difference between
+24:34
+these two architectures like our brains here and LLM brains.
+24:39
+You can look at it that way almost. Here's one thing that people found for example works pretty well in practice.
+24:45
+Especially if your tasks require reasoning, you can't expect the transformer to do too much reasoning per token.
+24:52
+You have to really spread out the reasoning across more and more tokens. For example, you can't give a transformer
+24:57
+a very complicated question and expect it to get the answer in a single token. There's just not enough time for it. "These transformers need tokens to
+25:04
+think," I like to say sometimes. This is some of the things that work well, you may for example have a few-shot prompt that
+25:10
+shows the transformer that it should show its work when it's answering question and if you give a few examples,
+25:17
+the transformer will imitate that template and it will just end up working out better in terms of its evaluation.
+25:24
+Additionally, you can elicit this behavior from the transformer by saying, let things step-by-step.
+25:29
+Because this conditions the transformer into showing its work and because
+25:34
+it snaps into a mode of showing its work, is going to do less computational work per token.
+25:40
+It's more likely to succeed as a result because it's making slower reasoning over time.
+25:46
+Here's another example, this one is called self-consistency. We saw that we had the ability
+Ensemble multiple attempts
+25:51
+to start writing and then if it didn't work out, I can try again and I can try multiple times
+25:56
+and maybe select the one that worked best. In these approaches,
+26:02
+you may sample not just once, but you may sample multiple times and then have some process for finding
+26:07
+the ones that are good and then keeping just those samples or doing a majority vote or something like that. Basically these transformers in the process as
+26:14
+they predict the next token, just like you, they can get unlucky and they could sample a not a very good
+26:19
+token and they can go down like a blind alley in terms of reasoning. Unlike you, they cannot recover from that.
+26:27
+They are stuck with every single token they sample and so they will continue the sequence, even if they know that this sequence is not going to work out.
+26:34
+Give them the ability to look back, inspect or try to basically sample around it.
+26:40
+Here's one technique also, it turns out that actually LLMs, they know when they've screwed up,
+Ask for reflection
+26:47
+so as an example, say you ask the model to generate a poem that does not
+26:52
+rhyme and it might give you a poem, but it actually rhymes. But it turns out that especially for the bigger models like GPT-4,
+26:58
+you can just ask it "did you meet the assignment?" Actually GPT-4 knows very well that it did not meet the assignment.
+27:04
+It just got unlucky in its sampling. It will tell you, "No, I didn't actually meet the assignment here. Let me try again."
+27:10
+But without you prompting it it doesn't know to revisit and so on.
+27:17
+You have to make up for that in your prompts, and you have to get it to check, if you don't ask it to check,
+27:23
+its not going to check by itself it's just a token simulator.
+27:28
+I think more generally, a lot of these techniques fall into the bucket of what I would say recreating our System 2.
+27:34
+You might be familiar with the System 1 and System 2 thinking for humans. System 1 is a fast automatic process and I
+27:40
+think corresponds to an LLM just sampling tokens. System 2 is the slower deliberate
+27:46
+planning part of your brain. This is a paper actually from
+27:51
+just last week because this space is pretty quickly evolving, it's called Tree of Thought.
+27:56
+The authors of this paper proposed maintaining multiple completions for any given prompt
+28:02
+and then they are also scoring them along the way and keeping the ones that are going well if that makes sense.
+28:08
+A lot of people are really playing around with prompt engineering
+28:13
+to basically bring back some of these abilities that we have in our brain for LLMs.
+28:19
+Now, one thing I would like to note here is that this is not just a prompt. This is actually prompts that are together
+28:25
+used with some Python Glue code because you actually have to maintain multiple prompts and you also have to do
+28:30
+some tree search algorithm here to figure out which prompts to expand, etc. It's a symbiosis of Python Glue code and
+28:38
+individual prompts that are called in a while loop or in a bigger algorithm. I also think there's a really cool
+28:43
+parallel here to AlphaGo. AlphaGo has a policy for placing the next stone when it plays go,
+28:48
+and its policy was trained originally by imitating humans. But in addition to this policy,
+28:54
+it also does Monte Carlo Tree Search. Basically, it will play out a number of possibilities in its head and evaluate all of
+29:00
+them and only keep the ones that work well. I think this is an equivalent of AlphaGo but for text if that makes sense.
+29:08
+Just like Tree of Thought, I think more generally people are starting to really explore
+29:13
+more general techniques of not just the simple question-answer prompts, but something that looks a lot more like
+29:19
+Python Glue code stringing together many prompts. On the right, I have an example from this paper called React where they
+29:25
+structure the answer to a prompt as a sequence of thought-action-observation,
+29:32
+thought-action-observation, and it's a full rollout and a thinking process to answer the query.
+29:38
+In these actions, the model is also allowed to tool use. On the left, I have an example of AutoGPT.
+29:45
+Now AutoGPT by the way is a project that I think got a lot of hype recently,
+29:51
+but I think I still find it inspirationally interesting. It's a project that allows an LLM to keep
+29:58
+the task list and continue to recursively break down tasks. I don't think this currently works very well and I would
+30:04
+not advise people to use it in practical applications. I just think it's something to generally take inspiration
+30:09
+from in terms of where this is going, I think over time. That's like giving our model System 2 thinking.
+30:16
+The next thing I find interesting is, this following serve I would say almost psychological quirk of LLMs,
+30:23
+is that LLMs don't want to succeed, they want to imitate. You want to succeed, and you should ask for it.
+30:31
+What I mean by that is, when transformers are trained, they have training sets and there can be
+30:38
+an entire spectrum of performance qualities in their training data. For example, there could be some kind of a prompt
+30:43
+for some physics question or something like that, and there could be a student's solution that is completely wrong but there can also be an expert
+30:49
+answer that is extremely right. Transformers can't tell the difference between low,
+30:54
+they know about low-quality solutions and high-quality solutions, but by default, they want to imitate all of
+30:59
+it because they're just trained on language modeling. At test time, you actually have to ask for a good performance.
+31:06
+In this example in this paper, they tried various prompts. Let's think step-by-step was very powerful
+31:13
+because it spread out the reasoning over many tokens. But what worked even better is, let's work this out in a step-by-step way
+31:19
+to be sure we have the right answer. It's like conditioning on getting the right answer, and this actually makes the transformer work
+31:25
+better because the transformer doesn't have to now hedge its probability mass on low-quality solutions,
+31:31
+as ridiculous as that sounds. Basically, feel free to ask for a strong solution.
+31:37
+Say something like, you are a leading expert on this topic. Pretend you have IQ 120, etc. But don't try to ask for too much IQ because if
+31:44
+you ask for IQ 400, you might be out of data distribution, or even worse, you could be in data distribution for
+31:51
+something like sci-fi stuff and it will start to take on some sci-fi, or like roleplaying or something like that.
+31:56
+You have to find the right amount of IQ. I think it's got some U-shaped curve there.
+32:02
+Next up, as we saw when we are trying to solve problems, we know what we are good at and what we're not good at,
+32:09
+and we lean on tools computationally. You want to do the same potentially with your LLMs.
+Tool use / Plugins
+32:15
+In particular, we may want to give them calculators, code interpreters,
+32:20
+and so on, the ability to do search, and there's a lot of techniques for doing that.
+32:27
+One thing to keep in mind, again, is that these transformers by default may not know what they don't know.
+32:32
+You may even want to tell the transformer in a prompt you are not very good at mental arithmetic. Whenever you need to do very large number addition,
+32:40
+multiplication, or whatever, instead, use this calculator. Here's how you use the calculator, you use this token combination, etc.
+32:46
+You have to actually spell it out because the model by default doesn't know what it's good at or not good at, necessarily, just like you and I might be.
+32:54
+Next up, I think something that is very interesting is we went from a world that was retrieval only all the way,
+33:02
+the pendulum has swung to the other extreme where its memory only in LLMs. But actually, there's this entire space in-between of
+33:08
+these retrieval-augmented models and this works extremely well in practice. As I mentioned, the context window of
+33:14
+a transformer is its working memory. If you can load the working memory with any information that is relevant to the task,
+33:21
+the model will work extremely well because it can immediately access all that memory. I think a lot of people are really interested
+33:28
+in basically retrieval-augment degeneration. On the bottom, I have an example of LlamaIndex which is
+33:35
+one data connector to lots of different types of data. You can index all
+33:41
+of that data and you can make it accessible to LLMs. The emerging recipe there is you take relevant documents,
+33:47
+you split them up into chunks, you embed all of them, and you basically get embedding vectors that represent that data.
+33:53
+You store that in the vector store and then at test time, you make some kind of a query to your vector store and you fetch chunks that
+34:00
+might be relevant to your task and you stuff them into the prompt and then you generate. This can work quite well in practice.
+34:06
+This is, I think, similar to when you and I solve problems. You can do everything from your memory and
+34:11
+transformers have very large and extensive memory, but also it really helps to reference some primary documents.
+34:17
+Whenever you find yourself going back to a textbook to find something, or whenever you find yourself going back to documentation of the library to look something up,
+34:25
+transformers definitely want to do that too. You have some memory over how
+34:30
+some documentation of the library works but it's much better to look it up. The same applies here.
+34:35
+Next, I wanted to briefly talk about constraint prompting. I also find this very interesting.
+34:41
+This is basically techniques for forcing a certain template in the outputs of LLMs.
+34:50
+Guidance is one example from Microsoft actually. Here we are enforcing that the output from the LLM will be JSON.
+34:57
+This will actually guarantee that the output will take on this form because they go in and they mess with the probabilities of
+35:03
+all the different tokens that come out of the transformer and they clamp those tokens and then the transformer is only filling in the blanks here,
+35:09
+and then you can enforce additional restrictions on what could go into those blanks. This might be really helpful, and I think
+35:15
+this constraint sampling is also extremely interesting. I also want to say
+35:20
+a few words about fine tuning. It is the case that you can get really far with prompt engineering, but it's also possible to
+35:27
+think about fine tuning your models. Now, fine tuning models means that you are actually going to change the weights of the model.
+35:33
+It is becoming a lot more accessible to do this in practice, and that's because of a number of techniques that have been
+35:39
+developed and have libraries for very recently. So for example parameter efficient fine tuning techniques like Laura,
+35:46
+make sure that you're only training small, sparse pieces of your model. So most of the model is kept clamped at
+35:53
+the base model and some pieces of it are allowed to change and this still works pretty well empirically and makes
+35:58
+it much cheaper to tune only small pieces of your model. It also means that because most of your model is clamped,
+36:05
+you can use very low precision inference for computing those parts because you are not going to be updated by
+36:10
+gradient descent and so that makes everything a lot more efficient as well. And in addition, we have a number of open source, high-quality base models.
+36:17
+Currently, as I mentioned, I think LLaMa is quite nice, although it is not commercially licensed, I believe right now.
+36:23
+Some things to keep in mind is that basically fine tuning is a lot more technically involved.
+36:29
+It requires a lot more, I think, technical expertise to do right. It requires human data contractors for
+36:34
+datasets and/or synthetic data pipelines that can be pretty complicated. This will definitely slow down
+36:40
+your iteration cycle by a lot, and I would say on a high level SFT is achievable because you're continuing
+36:47
+the language modeling task. It's relatively straightforward, but RLHF, I would say is very much research territory
+36:53
+and is even much harder to get to work, and so I would probably not advise that someone just tries to roll their own RLHF of implementation.
+37:00
+These things are pretty unstable, very difficult to train, not something that is, I think, very beginner friendly right now,
+37:06
+and it's also potentially likely also to change pretty rapidly still.
+37:11
+So I think these are my default recommendations right now. I would break up your task into two major parts.
+Default recommendations
+37:18
+Number 1, achieve your top performance, and Number 2, optimize your performance in that order.
+37:23
+Number 1, the best performance will currently come from GPT-4 model. It is the most capable of all by far.
+37:29
+Use prompts that are very detailed. They have lots of task content, relevant information and instructions.
+37:36
+Think along the lines of what would you tell a task contractor if they can't email you back, but then also keep in mind that a task contractor is a
+37:43
+human and they have inner monologue and they're very clever, etc. LLMs do not possess those qualities.
+37:48
+So make sure to think through the psychology of the LLM almost and cater prompts to that.
+37:54
+Retrieve and add any relevant context and information to these prompts. Basically refer to a lot of
+38:01
+the prompt engineering techniques. Some of them I've highlighted in the slides above, but also this is a very large space and I would
+38:07
+just advise you to look for prompt engineering techniques online. There's a lot to cover there.
+38:13
+Experiment with few-shot examples. What this refers to is, you don't just want to tell, you want to show whenever it's possible.
+38:19
+So give it examples of everything that helps it really understand what you mean if you can.
+38:25
+Experiment with tools and plug-ins to offload tasks that are difficult for LLMs natively,
+38:30
+and then think about not just a single prompt and answer, think about potential chains and reflection and how you glue
+38:36
+them together and how you can potentially make multiple samples and so on. Finally, if you think you've squeezed
+38:42
+out prompt engineering, which I think you should stick with for a while, look at some potentially
+38:48
+fine tuning a model to your application, but expect this to be a lot more slower in the vault and then
+38:54
+there's an expert fragile research zone here and I would say that is RLHF, which currently does work a bit
+39:00
+better than SFT if you can get it to work. But again, this is pretty involved, I would say. And to optimize your costs,
+39:06
+try to explore lower capacity models or shorter prompts and so on.
+39:12
+I also wanted to say a few words about the use cases in which I think LLMs are currently well suited for.
+39:18
+In particular, note that there's a large number of limitations to LLMs today, and so I would keep that
+39:24
+definitely in mind for all of your applications. Models, and this by the way could be an entire talk. So I don't have time to cover it in full detail.
+39:30
+Models may be biased, they may fabricate, hallucinate information, they may have reasoning errors, they may struggle in entire classes of applications,
+39:38
+they have knowledge cut-offs, so they might not know any information above, say, September, 2021.
+39:43
+They are susceptible to a large range of attacks which are coming out on Twitter daily,
+39:48
+including prompt injection, jailbreak attacks, data poisoning attacks and so on. So my recommendation right now is
+39:54
+use LLMs in low-stakes applications. Combine them always with human oversight.
+40:00
+Use them as a source of inspiration and suggestions and think co-pilots, instead of completely autonomous agents
+40:05
+that are just like performing a task somewhere. It's just not clear that the models are there right now.
+40:11
+So I wanted to close by saying that GPT-4 is an amazing artifact. I'm very thankful that it exists, and it's beautiful.
+40:18
+It has a ton of knowledge across so many areas. It can do math, code and so on. And in addition, there's this
+40:24
+thriving ecosystem of everything else that is being built and incorporated into the ecosystem. Some of these things I've talked about,
+40:31
+and all of this power is accessible at your fingertips. So here's everything that's needed in terms of
+40:37
+code to ask GPT-4 a question, to prompt it, and get a response. In this case, I said,
+40:44
+can you say something to inspire the audience of Microsoft Build 2023? And I just punched this into Python and verbatim
+40:50
+GPT-4 said the following: And by the way, I did not know that they
+40:55
+used this trick in the keynote. So I thought I was being clever, but it is really good at this.
+41:02
+It says, ladies and gentlemen, innovators and trailblazers Microsoft Build 2023. Welcome to the gathering of brilliant
+41:08
+minds like no other, you are the architects of the future, the visionaries molding the digital realm
+41:13
+in which humanity thrives. Embrace the limitless possibilities of technologies and let your ideas soar as high as your imagination.
+41:20
+Together, let's create a more connected, remarkable, and inclusive world for generations to come. Get ready to unleash your creativity,
+41:27
+canvas the unknown, and turn dreams into reality. Your journey begins today!
+