WEBVTT 00:00:00.000 --> 00:00:04.679 Right now, whether you are using chat GPT to 00:00:04.679 --> 00:00:07.860 write an email, or relying on Google Translate 00:00:07.860 --> 00:00:10.980 to read a menu in Tokyo, or even watching an 00:00:10.980 --> 00:00:14.000 AI generate a hyper realistic video from a text 00:00:14.000 --> 00:00:16.800 prompt, you are interacting with a hidden, wildly 00:00:16.800 --> 00:00:18.859 powerful engine. Yeah, absolutely. It's called 00:00:18.859 --> 00:00:20.940 the transformer. So welcome to the deep dive. 00:00:21.120 --> 00:00:24.940 Thanks for having me. Today, our mission is to 00:00:24.940 --> 00:00:27.699 really look under the hood of this architecture. 00:00:27.920 --> 00:00:30.960 We are pulling from an incredibly comprehensive, 00:00:31.660 --> 00:00:34.140 frankly kind of dense Wikipedia article. Oh, 00:00:34.259 --> 00:00:36.880 very dense. Yeah, detailing the history and the 00:00:36.880 --> 00:00:39.359 mechanics of the transformer. Our goal today 00:00:39.359 --> 00:00:42.700 is to skip the overwhelming jargon. bypass the 00:00:42.700 --> 00:00:46.340 hype and give you a real shortcut to understanding 00:00:46.340 --> 00:00:48.920 the core technology driving this massive artificial 00:00:48.920 --> 00:00:50.740 intelligence boom we're living through. Because 00:00:50.740 --> 00:00:52.880 understanding how these tools actually function, 00:00:53.280 --> 00:00:55.700 like the literal mechanics underneath the interface, 00:00:56.100 --> 00:00:58.560 it gives you a massive edge in a world that is 00:00:58.560 --> 00:01:00.439 just suffering from severe information overload 00:01:00.439 --> 00:01:03.439 right now. Oh, 100%. Once you grasp the underlying 00:01:03.439 --> 00:01:06.620 logic of this specific architecture, I mean, 00:01:06.920 --> 00:01:09.819 an entire decade of rapid, seemingly magical 00:01:09.819 --> 00:01:12.900 technological innovation suddenly makes perfect, 00:01:13.120 --> 00:01:15.859 rational sense. OK, let's unpack this. Because 00:01:15.859 --> 00:01:18.019 to really appreciate the brilliance of the transformer, 00:01:18.079 --> 00:01:20.219 we first have to understand the sheer frustration 00:01:20.219 --> 00:01:22.420 of the technology that came before it. Right. 00:01:22.700 --> 00:01:26.659 The AI world before 2017 was not moving at this 00:01:26.659 --> 00:01:28.560 breakneck speed. No, not at all. And there was 00:01:28.560 --> 00:01:31.329 a very specific physical bottleneck holding everything 00:01:31.329 --> 00:01:34.890 back. Yeah, so the era before 2017 was completely 00:01:34.890 --> 00:01:37.069 dominated by recurrent neural networks. More 00:01:37.069 --> 00:01:40.969 RNNs. Exactly, RNNs. And a specialized variation 00:01:40.969 --> 00:01:43.329 called long short -term memory networks. LSTMs, 00:01:43.409 --> 00:01:46.530 right? Yes, LSTMs. And these concepts, they were 00:01:46.530 --> 00:01:49.069 originally conceptualized way back in the 1990s. 00:01:49.069 --> 00:01:51.689 Wow, okay. Yeah, they were the standard for sequence 00:01:51.689 --> 00:01:54.609 modeling. So things like translating a sentence 00:01:54.609 --> 00:01:58.189 from French to English. But the fatal flaw built 00:01:58.189 --> 00:02:00.769 into their very design was sequential processing. 00:02:00.959 --> 00:02:03.019 Meaning what exactly? Meaning they operated on 00:02:03.019 --> 00:02:05.219 one token or you know one word at a time strictly 00:02:05.219 --> 00:02:07.480 from the first to the last. I always picture 00:02:07.480 --> 00:02:11.180 this like trying to read a dense complicated 00:02:11.180 --> 00:02:14.250 novel through a tiny cocktail straw. That is 00:02:14.250 --> 00:02:16.930 a great analogy. Word by word, right? You look 00:02:16.930 --> 00:02:18.349 through the straw and you see the word they, 00:02:18.669 --> 00:02:21.289 then you move it and see quick, and by the time 00:02:21.289 --> 00:02:23.930 you get to the very end of a long, twisting sentence, 00:02:24.330 --> 00:02:26.490 you've completely forgotten how it started. Yeah, 00:02:26.530 --> 00:02:29.270 and the technical term for that exact phenomenon 00:02:29.270 --> 00:02:32.610 is the vanishing gradient problem. The vanishing 00:02:32.610 --> 00:02:35.580 gradient. Right. Because an RNN processes the 00:02:35.580 --> 00:02:38.180 input sequentially, updating its internal state 00:02:38.180 --> 00:02:41.240 step by step, it has to compress all that historical 00:02:41.240 --> 00:02:44.759 context into a single fixed size output vector. 00:02:45.060 --> 00:02:47.139 Right. OK. So by the time the model reaches the 00:02:47.139 --> 00:02:50.099 end of a long paragraph, its internal state lacks 00:02:50.099 --> 00:02:52.879 any precise extractable information about the 00:02:52.879 --> 00:02:55.319 words at the very beginning. The mathematical 00:02:55.319 --> 00:02:58.560 signal just dilutes and, well, vanishes. So if 00:02:58.560 --> 00:03:01.120 I feed it a massive document to translate, it's 00:03:01.120 --> 00:03:03.020 essentially running out of memory by the final 00:03:02.990 --> 00:03:05.090 punctuation mark. Exactly the information is 00:03:05.090 --> 00:03:08.030 completely overwritten and smoothed out and you 00:03:08.030 --> 00:03:10.710 know researchers were highly aware of this bottleneck. 00:03:10.729 --> 00:03:13.330 They knew it was a problem. Oh yeah early models 00:03:13.330 --> 00:03:16.030 like something called the RNN search model they 00:03:16.030 --> 00:03:18.930 attempted to fix this by introducing early attention 00:03:18.930 --> 00:03:21.469 mechanisms. Okay so attention was around before 00:03:21.469 --> 00:03:24.509 the transformer. It was. This early attention 00:03:24.509 --> 00:03:27.110 allowed the model to maintain a kind of memory 00:03:27.110 --> 00:03:31.180 bank of the source sentence and sort of search 00:03:31.180 --> 00:03:32.919 through it while generating the translation. 00:03:33.419 --> 00:03:35.759 But the underlying architecture was still built 00:03:35.759 --> 00:03:38.819 on recurrent networks. And that meant it suffered 00:03:38.819 --> 00:03:41.280 from the ultimate sin of modern computing. It 00:03:41.280 --> 00:03:43.419 could not be parallelized. Right. Because step 00:03:43.419 --> 00:03:45.800 two physically cannot happen until step one is 00:03:45.800 --> 00:03:48.819 finished. Exactly. You can't split the job across 00:03:48.819 --> 00:03:52.379 a massive farm of modern graphic processing units, 00:03:52.460 --> 00:03:55.020 you know, GPUs. Because the GPUs are just sitting 00:03:55.020 --> 00:03:57.300 there waiting for the previous word to be processed. 00:03:57.580 --> 00:04:00.430 And that sequential lock is what kept AI training 00:04:00.430 --> 00:04:03.610 so slow. The hardware was fully capable of doing 00:04:03.610 --> 00:04:05.969 millions of calculations at the same time, but 00:04:05.969 --> 00:04:09.189 the RNN software was forcing it to work in single 00:04:09.189 --> 00:04:11.590 file. Which brings us to the breakthrough. Yes. 00:04:11.710 --> 00:04:14.349 The sheer limitation of reading sequentially 00:04:14.349 --> 00:04:18.269 led some researchers to propose a radical hypothesis. 00:04:18.850 --> 00:04:22.089 What if we stop reading in order? Right. What 00:04:22.089 --> 00:04:24.089 if we just read the entire book all at once? 00:04:24.329 --> 00:04:27.569 And that question birthed the 2017 Google paper 00:04:27.569 --> 00:04:31.259 titled Attention is all you need, it proposed 00:04:31.259 --> 00:04:34.439 dropping recurrence completely. I love the trivia 00:04:34.439 --> 00:04:36.360 our source includes about this paper, by the 00:04:36.360 --> 00:04:38.899 way. Oh, a family connection. Yes. One of the 00:04:38.899 --> 00:04:42.579 authors, Jacob Ouzkarait, he suspected that attention 00:04:42.579 --> 00:04:45.439 without any recurrence would be enough for translation. 00:04:45.759 --> 00:04:49.269 Right. And this idea went so hard against the 00:04:49.269 --> 00:04:51.490 conventional wisdom of the time that even his 00:04:51.490 --> 00:04:53.610 father, Hans Uuskarait... Who happens to be a 00:04:53.610 --> 00:04:55.829 very famous computational linguist, by the way. 00:04:55.910 --> 00:04:57.930 Right. Even his dad was highly skeptical that 00:04:57.930 --> 00:04:59.550 you could just drop a recurrence entirely. It 00:04:59.550 --> 00:05:01.490 just sounded absurd. It really was a massive 00:05:01.490 --> 00:05:03.569 leap of faith. I mean, to abandon the only method 00:05:03.569 --> 00:05:05.930 that had reliably worked for sequence modeling, 00:05:06.329 --> 00:05:09.170 the paper proposed an entirely new model that 00:05:09.170 --> 00:05:11.889 relied purely on an attention mechanism to draw 00:05:11.889 --> 00:05:14.769 global dependencies between input and output. 00:05:15.969 --> 00:05:18.649 What's fascinating here is the specific mathematical 00:05:18.649 --> 00:05:21.430 engine they invented to do this. Called scale 00:05:21.430 --> 00:05:24.089 .productAttention. Exactly. This is the absolute 00:05:24.089 --> 00:05:25.970 core of the engine, so let's really break this 00:05:25.970 --> 00:05:28.670 down. From what I understand, the model learns 00:05:28.670 --> 00:05:32.029 three specific weight matrices for every single 00:05:32.029 --> 00:05:34.910 token. Right. The query, the key, and the value. 00:05:35.430 --> 00:05:38.730 Q, K, and V. Yeah, so for every token in a sequence, 00:05:38.930 --> 00:05:41.129 the model generates those three vectors, and 00:05:41.129 --> 00:05:43.750 it uses them to mathematically ask a very specific 00:05:43.750 --> 00:05:46.449 question. It asks, how much attention should 00:05:46.449 --> 00:05:49.129 this word pay to every other word in the sequence? 00:05:49.589 --> 00:05:52.589 Instead of a filing cabinet with like exact labels. 00:05:52.709 --> 00:05:54.589 I like to think of it like asking a giant crowd 00:05:54.589 --> 00:05:56.689 for a highly specific recommendation. Oh, I like 00:05:56.689 --> 00:05:59.009 that. How does that work? So, Curie is me shouting 00:05:59.009 --> 00:06:02.110 to the crowd, who here likes 1980s sci -fi movies? 00:06:02.209 --> 00:06:04.790 Okay. And the keys are the graphic t -shirts 00:06:04.790 --> 00:06:06.589 everyone in the crowd is wearing. Right, right. 00:06:06.649 --> 00:06:08.870 The model uses a dot product, which is basically 00:06:08.870 --> 00:06:11.250 a mathematical measurement of similarity. Yes, 00:06:11.370 --> 00:06:13.709 it measures alignment. Right. To calculate how 00:06:13.709 --> 00:06:15.750 closely everyone's t -shirt matches my question. 00:06:16.470 --> 00:06:18.230 So, someone is wearing a retro Blade Runner shirt. 00:06:18.519 --> 00:06:20.920 the math gives them a very high attention weight. 00:06:21.379 --> 00:06:24.220 Exactly. Then the value is the actual movie recommendation 00:06:24.220 --> 00:06:26.720 that person gives me. So the system aggregates 00:06:26.720 --> 00:06:29.620 all the advice from the crowd, prioritizing the 00:06:29.620 --> 00:06:32.000 people with the highest attention weights. That 00:06:32.000 --> 00:06:36.079 is a highly accurate way to visualize a dot product. 00:06:36.439 --> 00:06:39.240 It calculates the alignment between two vectors. 00:06:39.759 --> 00:06:42.259 A higher dot product means the query and the 00:06:42.259 --> 00:06:45.319 key are highly aligned, so the model pulls more 00:06:45.319 --> 00:06:47.759 of that specific value into its understanding 00:06:47.759 --> 00:06:50.379 of the word. Wow. And the transformer doesn't 00:06:50.379 --> 00:06:53.579 just do this once. It uses multi -head attention. 00:06:53.740 --> 00:06:56.500 Multi -head attention. Right. It runs this exact 00:06:56.500 --> 00:06:58.980 crowdsourcing process you just described multiple 00:06:58.980 --> 00:07:01.600 times simultaneously through different lenses. 00:07:01.860 --> 00:07:04.120 So it's asking different types of questions to 00:07:04.120 --> 00:07:06.060 the crowd at the exact same time. Precisely. 00:07:06.300 --> 00:07:08.980 In one attention head, the query might be looking 00:07:08.980 --> 00:07:11.680 for syntactic relationships, you know, like how 00:07:11.680 --> 00:07:14.680 pronouns relate to their nouns. While in another 00:07:14.680 --> 00:07:17.500 head... operating in parallel, the query might 00:07:17.500 --> 00:07:20.379 be looking for semantic meaning, like how a verb 00:07:20.379 --> 00:07:23.120 connects to its direct object. Or maybe another 00:07:23.120 --> 00:07:24.680 head is just paying attention to the punctuation. 00:07:24.879 --> 00:07:27.600 Exactly. And then it concatenates all these different 00:07:27.600 --> 00:07:30.000 perspectives together. This is what allows the 00:07:30.000 --> 00:07:33.480 model to capture incredibly complex, long -range 00:07:33.480 --> 00:07:36.439 dependencies across the entire text instantly. 00:07:37.019 --> 00:07:38.660 OK, wait. I need to push back logically here. 00:07:38.720 --> 00:07:40.839 Go for it. If the model is reading the whole 00:07:40.839 --> 00:07:44.259 sentence simultaneously, in pure parallel. How 00:07:44.259 --> 00:07:46.220 does it actually know the order of the words? 00:07:46.300 --> 00:07:49.040 Ah, that's the big question. Because to a purely 00:07:49.040 --> 00:07:51.980 parallel system, aren't the sentences man bites 00:07:51.980 --> 00:07:55.120 dog and dog bites man just the exact same collection 00:07:55.120 --> 00:07:57.839 of words? Without a sequence, how does it know 00:07:57.839 --> 00:08:00.019 the story? You hit the nail on the head. The 00:08:00.019 --> 00:08:02.699 creators recognized that pure attention has zero 00:08:02.699 --> 00:08:05.579 built -in sense of sequence. So to fix this, 00:08:05.779 --> 00:08:08.680 they introduced positional encoding. Before the 00:08:08.680 --> 00:08:11.060 text even hits that massive attention mechanism, 00:08:11.500 --> 00:08:13.920 they mathematically inject a unique positional 00:08:13.920 --> 00:08:16.680 signature into the token itself. So they basically 00:08:16.680 --> 00:08:19.139 stamp every single word with GPS coordinates. 00:08:19.480 --> 00:08:21.480 That's a perfect way to put it. The original 00:08:21.480 --> 00:08:24.199 paper used complex sine and cosine functions 00:08:24.199 --> 00:08:27.180 of different frequencies to create a fixed size 00:08:27.180 --> 00:08:29.720 vector representing the absolute and relative 00:08:29.720 --> 00:08:32.659 position of the token. And newer models. Newer 00:08:32.659 --> 00:08:35.899 models use alternative methods like ROPEE. That 00:08:35.899 --> 00:08:38.519 stands for rotary positional embedding. ROPEE. 00:08:38.679 --> 00:08:41.539 Which literally rotates the vector representation 00:08:41.539 --> 00:08:44.240 by a specific angle based on its position in 00:08:44.240 --> 00:08:46.960 the sequence. But the underlying result is the 00:08:46.960 --> 00:08:50.000 same. The model processes everything at once, 00:08:50.440 --> 00:08:52.700 but the mathematical signature of the data tells 00:08:52.700 --> 00:08:55.259 the model where everything is located relative 00:08:55.259 --> 00:08:57.759 to everything else. Here's where it gets really 00:08:57.759 --> 00:09:00.200 interesting, because when we keep saying words, 00:09:00.620 --> 00:09:03.639 we're using a human term. Right. The transformer 00:09:03.639 --> 00:09:05.259 doesn't actually know what an English word is. 00:09:05.340 --> 00:09:07.360 No, it has no concept of a word. It goes through 00:09:07.360 --> 00:09:10.039 a process called tokenization. Right. The architecture 00:09:10.039 --> 00:09:12.659 natively consists of operations over numbers, 00:09:13.340 --> 00:09:15.620 matrix multiplications, and dot products. It 00:09:15.620 --> 00:09:18.279 takes the input text, chops it up into core segments, 00:09:18.480 --> 00:09:20.960 and then a tokenizer breaks it down further into 00:09:20.960 --> 00:09:23.600 tokens from a strictly fixed vocabulary. And 00:09:23.600 --> 00:09:25.799 a token isn't always a whole word. Exactly. A 00:09:25.799 --> 00:09:28.179 token could be a whole word, but often it's just 00:09:28.179 --> 00:09:31.460 a chunk of a word. a subword like ing or pre, 00:09:32.139 --> 00:09:35.399 those tokens are then mapped to integer identifiers 00:09:35.399 --> 00:09:38.399 and finally embedded into high dimensional vectors. 00:09:38.899 --> 00:09:41.279 And what comes out the other end is equally wild, 00:09:41.799 --> 00:09:44.360 the unembedding layer. Yes. Because once the 00:09:44.360 --> 00:09:47.000 transformer has done all its massive multi -headed 00:09:47.000 --> 00:09:50.340 parallel processing, you have these rich, dense, 00:09:50.679 --> 00:09:53.629 contextualized vectors. but you need to turn 00:09:53.629 --> 00:09:55.450 them back into something a human can actually 00:09:55.450 --> 00:09:57.610 read. Right, so it passes those final vectors 00:09:57.610 --> 00:10:00.409 through a linear softmax layer. It takes the 00:10:00.409 --> 00:10:03.169 vector, multiplies it by a weight matrix of shape 00:10:03.169 --> 00:10:05.509 dm by the absolute size of the vocabulary to 00:10:05.509 --> 00:10:08.149 produce logits, and then applies the softmax 00:10:08.149 --> 00:10:10.350 function to normalize those. Okay wait, I'm gonna 00:10:10.350 --> 00:10:12.230 stop you right there because that is pure textbook 00:10:12.230 --> 00:10:13.889 territory. Oh fair enough, I get carried away. 00:10:14.039 --> 00:10:15.759 Let's unpack that math into something a little 00:10:15.759 --> 00:10:18.559 more visual. What is a logit and what is softmax? 00:10:18.820 --> 00:10:21.960 Okay, okay. Imagine a giant scoreboard that contains 00:10:21.960 --> 00:10:25.600 every single token in the AI's entire vocabulary. 00:10:25.840 --> 00:10:29.360 Millions of subwords. Exactly. The model assigns 00:10:29.360 --> 00:10:32.700 a raw mathematical score to every single item 00:10:32.700 --> 00:10:35.600 on that scoreboard based on its processing. Those 00:10:35.600 --> 00:10:38.769 raw scores? Those are the logits. But raw scores 00:10:38.769 --> 00:10:41.029 are messy, right? Yeah. Like one word might have 00:10:41.029 --> 00:10:43.190 a score of 5 ,000 and another might be negative 00:10:43.190 --> 00:10:46.419 400. Right. which is incredibly hard to interpret. 00:10:46.600 --> 00:10:49.919 So it runs all those raw scores through a mathematical 00:10:49.919 --> 00:10:53.320 filter called the softmax function. Softmax essentially 00:10:53.320 --> 00:10:56.039 forces all those wild numbers to fit into a neat 00:10:56.039 --> 00:10:58.840 pie chart of percentages that perfectly add up 00:10:58.840 --> 00:11:02.259 to 100%. Oh, I see. So it converts dense linear 00:11:02.259 --> 00:11:05.419 algebra into a giant list of probabilities, saying, 00:11:05.480 --> 00:11:08.559 you know, I am 99 % sure the next token should 00:11:08.559 --> 00:11:11.539 be Apple. And maybe 0 .5 % sure it should be 00:11:11.539 --> 00:11:15.679 banana, and 0 .001 % sure it's xylophone. Exactly. 00:11:15.820 --> 00:11:19.320 It creates a clear, actionable probability distribution. 00:11:19.700 --> 00:11:21.980 It ranks the vocabulary, picks the winner, and 00:11:21.980 --> 00:11:23.460 that's the word that appears on your screen. 00:11:23.519 --> 00:11:25.679 You got it. So now that we understand the engine, 00:11:26.539 --> 00:11:29.059 massive parallel processing using queries, keys, 00:11:29.139 --> 00:11:31.559 and values stamped with positional encodings 00:11:31.559 --> 00:11:33.879 and filtered through a probability scoreboard. 00:11:34.080 --> 00:11:36.720 Right. How do different companies actually build 00:11:36.720 --> 00:11:40.000 cars around this engine? Our source outlines 00:11:40.000 --> 00:11:42.759 a transformer family tree. But what really matters 00:11:42.759 --> 00:11:44.960 here is the problem you are trying to solve, 00:11:45.419 --> 00:11:47.399 right? You can't use the exact same architecture 00:11:47.399 --> 00:11:50.700 to solve every problem. Exactly. The original 00:11:50.700 --> 00:11:54.100 2017 transformer was an encoder -decoder model. 00:11:54.820 --> 00:11:57.080 But researchers quickly realized you could rip 00:11:57.080 --> 00:11:58.879 those two parts away from each other depending 00:11:58.879 --> 00:12:01.159 on the specific task. OK, give me an example. 00:12:01.399 --> 00:12:04.149 Well, if your goal is pure comprehension, Say 00:12:04.149 --> 00:12:06.090 you are building a search engine that needs to 00:12:06.090 --> 00:12:08.289 deeply understand the context of a document. 00:12:08.789 --> 00:12:11.210 You build an encoder -only model, like Google's 00:12:11.210 --> 00:12:14.070 BERT. Because an encoder -only model processes 00:12:14.070 --> 00:12:16.750 the entire input all at once, looking forward 00:12:16.750 --> 00:12:19.509 and backward simultaneously. Exactly. I always 00:12:19.509 --> 00:12:21.370 compare training a model like Bert to playing 00:12:21.370 --> 00:12:24.309 a massive high -speed game of Mad Libs. Oh, yeah. 00:12:24.450 --> 00:12:26.690 You take a huge amount of text, you randomly 00:12:26.690 --> 00:12:30.169 blank out certain words, you mask them, and you 00:12:30.169 --> 00:12:32.990 force the AI to guess the blank spaces based 00:12:32.990 --> 00:12:35.409 on the surrounding context. It learns the deep 00:12:35.409 --> 00:12:37.370 structure of language by constantly trying to 00:12:37.370 --> 00:12:39.730 fill in the blanks. Yeah, the loss function measures 00:12:39.730 --> 00:12:42.629 how well it guesses those mass tokens, and it 00:12:42.629 --> 00:12:44.809 subtly adjusts its internal weights after every 00:12:44.809 --> 00:12:47.700 single guess. Fascinating. But now, if your goal 00:12:47.700 --> 00:12:50.279 is completely different, if you want an AI to 00:12:50.279 --> 00:12:53.399 actually write an essay or generate code, an 00:12:53.399 --> 00:12:56.100 encoder -only model isn't the right tool. You 00:12:56.100 --> 00:12:58.919 need a decoder -only model. And this is the GPT 00:12:58.919 --> 00:13:01.519 series, Generative Pre -trained Transformer. 00:13:01.679 --> 00:13:03.980 Which brings up a massive logical puzzle for 00:13:03.980 --> 00:13:06.620 me. What's that? If the entire point of a transformer 00:13:06.620 --> 00:13:09.460 is that it reads everything in parallel all at 00:13:09.460 --> 00:13:12.419 once, how does a decoder model write a story 00:13:12.419 --> 00:13:15.639 without cheating? If it can see the whole context 00:13:15.639 --> 00:13:18.179 simultaneously, wouldn't it look at the end of 00:13:18.179 --> 00:13:19.860 the sentence before it generates the beginning? 00:13:20.120 --> 00:13:22.659 That is the exact problem with pure parallel 00:13:22.659 --> 00:13:25.279 processing when generating text. To solve it, 00:13:25.379 --> 00:13:27.559 decoder -only models use something called causal 00:13:27.559 --> 00:13:30.059 masking. Causal masking. Right. When the model 00:13:30.059 --> 00:13:32.799 is predicting token number 5, it physically cannot 00:13:32.799 --> 00:13:36.399 pay attention to tokens 6, 7, or 8. The mask 00:13:36.399 --> 00:13:39.120 matrix literally sets the attention weights for 00:13:39.120 --> 00:13:42.429 any future tokens to negative infinity. The math 00:13:42.429 --> 00:13:45.570 forces them to be completely ignored. Wow, okay. 00:13:45.830 --> 00:13:48.009 So it effectively blinds the model to the future, 00:13:48.230 --> 00:13:51.549 forcing it to generate text strictly step -by 00:13:51.549 --> 00:13:54.230 -step. Autoregressively, yes. And then, if you 00:13:54.230 --> 00:13:56.990 want the best of both worlds, like translating 00:13:56.990 --> 00:14:00.169 a massive document where you need to deeply understand 00:14:00.169 --> 00:14:03.690 the entire source text before generating a completely 00:14:03.690 --> 00:14:06.049 new output sequence. Then you use an encoder 00:14:06.049 --> 00:14:08.850 -decoder mix, like the T5 models. Right. Looking 00:14:08.850 --> 00:14:12.070 at the timeline in our source, The speed of this 00:14:12.070 --> 00:14:14.570 evolution is just staggering. Unbelievable, really. 00:14:14.769 --> 00:14:17.970 In 2016, Google Translate revamped its system 00:14:17.970 --> 00:14:20.090 using an early sequence -to -sequence model. 00:14:20.289 --> 00:14:23.070 It took them nine months to develop, and it obliterated 00:14:23.070 --> 00:14:25.070 their statistical approach that had taken 10 00:14:25.070 --> 00:14:26.990 years to build. Ten years of work replaced in 00:14:26.990 --> 00:14:30.149 nine months. Yeah. Then bird arrives in 2018, 00:14:30.730 --> 00:14:33.149 GPT starts scaling up, and suddenly we have the 00:14:33.149 --> 00:14:36.809 explosion of chat GPT in late 2022. The boom 00:14:36.809 --> 00:14:39.610 was unprecedented. But it's important to note, 00:14:39.750 --> 00:14:42.710 it hit a severe physical wall very quickly because 00:14:42.710 --> 00:14:45.129 of the underlying math. Oh, right. Because the 00:14:45.129 --> 00:14:47.470 self -attention mechanism requires every single 00:14:47.470 --> 00:14:49.669 token to be compared against every other token. 00:14:50.110 --> 00:14:52.590 This means the compute time scales quadratically. 00:14:52.830 --> 00:14:55.330 Meaning, if you double the length of your prompt, 00:14:55.769 --> 00:14:57.970 it doesn't get twice as hard for the model to 00:14:57.970 --> 00:15:00.490 process. It gets four times harder. Exactly. 00:15:00.549 --> 00:15:03.129 If you try to feed a model an entire 500 page 00:15:03.129 --> 00:15:05.649 book, the computation becomes astronomically 00:15:05.649 --> 00:15:09.240 expensive. Yeah. engineers had to get incredibly 00:15:09.240 --> 00:15:12.340 creative to trip the math and make these tools 00:15:12.340 --> 00:15:15.139 efficient enough to, you know, run on your phone. 00:15:15.539 --> 00:15:17.259 Yeah, and the source outlined some brilliant 00:15:17.259 --> 00:15:20.659 workarounds. A major one is KV caching. How does 00:15:20.659 --> 00:15:23.440 that work? When a model like GPT is generating 00:15:23.440 --> 00:15:26.940 text autoregressively, word by word, The query 00:15:26.940 --> 00:15:29.500 vector changes at each step because it is actively 00:15:29.500 --> 00:15:32.960 looking for the new next word. But the key and 00:15:32.960 --> 00:15:34.899 value vectors for all the words it has already 00:15:34.899 --> 00:15:37.419 processed in the prompt, those are static. They 00:15:37.419 --> 00:15:40.669 don't change. So instead of throwing them away 00:15:40.669 --> 00:15:43.049 and recalculating the entire prompt from scratch 00:15:43.049 --> 00:15:45.730 for every single new syllable it spits out, KV 00:15:45.730 --> 00:15:48.230 caching just saves those computed vectors directly 00:15:48.230 --> 00:15:50.250 into memory. Exactly. It stores the historical 00:15:50.250 --> 00:15:53.309 context, massively reducing redundant math. And 00:15:53.309 --> 00:15:56.070 they paired the software trick with intense hardware 00:15:56.070 --> 00:15:58.269 optimizations like flash attention. Flash it. 00:15:58.490 --> 00:16:01.309 Yeah. Flash attention attacks the quadratic bottleneck 00:16:01.309 --> 00:16:04.649 by recognizing a physical limitation of GPUs. 00:16:04.830 --> 00:16:07.230 It's not just about doing the math. It's about 00:16:07.230 --> 00:16:09.759 the physical traffic jam. of moving data back 00:16:09.759 --> 00:16:12.740 and forth between the GPU's slow main memory 00:16:12.740 --> 00:16:15.940 and its super fast processing cache. It's like 00:16:15.940 --> 00:16:18.240 having the world's fastest chef in a kitchen 00:16:18.240 --> 00:16:20.639 but forcing him to walk down the street to the 00:16:20.639 --> 00:16:22.639 grocery store every time he needs a single onion. 00:16:23.000 --> 00:16:26.080 Exactly. Flash Attention reorganizes the algorithm 00:16:26.080 --> 00:16:28.860 so it performs matrix multiplications and blocks 00:16:28.860 --> 00:16:31.720 that fit perfectly within that fast local cache. 00:16:31.860 --> 00:16:33.919 That's so smart. It calculates the attention 00:16:33.919 --> 00:16:36.039 weights without constantly writing intermediate 00:16:36.039 --> 00:16:39.750 results back to the slow memory. It's a communication 00:16:39.750 --> 00:16:42.169 avoiding algorithm that drastically speeds up 00:16:42.169 --> 00:16:44.590 the physical processing time. I also have to 00:16:44.590 --> 00:16:46.669 raise a fascinating optimization mentioned in 00:16:46.669 --> 00:16:49.889 the notes. Speculative decoding. Oh, that's a 00:16:49.889 --> 00:16:52.070 good one. This is such a clever structural idea. 00:16:52.090 --> 00:16:55.110 You basically use a smaller, faster, much cheaper 00:16:55.110 --> 00:16:58.250 model to quickly guess the next few words in 00:16:58.250 --> 00:17:01.250 a sequence. Right. Like a junior assistant hurriedly 00:17:01.250 --> 00:17:04.170 writing a rough draft. Then you pass those guesses 00:17:04.170 --> 00:17:07.480 to the massive expensive model. because it's 00:17:07.480 --> 00:17:10.940 a transformer, the big model can read and verify 00:17:10.940 --> 00:17:14.539 all those guesses in parallel instantly. If the 00:17:14.539 --> 00:17:17.839 big boss agrees with the draft, you just saved 00:17:17.839 --> 00:17:21.279 a massive amount of generation time. If it disagrees, 00:17:21.619 --> 00:17:24.119 it just throws out the bad guesses. and computes 00:17:24.119 --> 00:17:27.240 the right answer itself. It trades parallel verification 00:17:27.240 --> 00:17:30.519 speed for autoregressive decoding time. It leans 00:17:30.519 --> 00:17:33.099 entirely into the exact strengths of the architecture. 00:17:33.700 --> 00:17:35.980 And if we connect this to the bigger picture, 00:17:36.500 --> 00:17:39.000 the realization that this architecture is incredibly 00:17:39.000 --> 00:17:41.400 flexible is really what launched the current 00:17:41.400 --> 00:17:43.420 era. Right, because the transformer didn't stay 00:17:43.420 --> 00:17:45.680 confined to text. Not at all. We saw the move 00:17:45.680 --> 00:17:48.339 to multimodality, treating everything like a 00:17:48.339 --> 00:17:50.900 language. Researchers realized that a transformer 00:17:50.900 --> 00:17:53.480 has no idea what a token actually represents. 00:17:53.700 --> 00:17:55.900 Like we said earlier, it only processes numbers. 00:17:56.420 --> 00:17:59.019 So what happens if a token isn't a subword? What 00:17:59.019 --> 00:18:01.240 if you take a high -resolution image, break it 00:18:01.240 --> 00:18:04.119 down into a grid of tiny square patches, turn 00:18:04.119 --> 00:18:06.859 those visual patches into vectors, and feed them 00:18:06.859 --> 00:18:09.000 into a transformer? You get vision transformers. 00:18:09.440 --> 00:18:12.799 This is the technology underlying incredible 00:18:12.799 --> 00:18:16.470 image generators, like Dali. or video generators 00:18:16.470 --> 00:18:19.730 like Sora. It treats groups of pixels the exact 00:18:19.730 --> 00:18:21.630 same way it treats syllables. And you can do 00:18:21.630 --> 00:18:23.829 the exact same thing with audio. Models like 00:18:23.829 --> 00:18:27.130 Whisper turn human speech into a visual spectrogram 00:18:27.130 --> 00:18:30.150 image, break that image into patches, and process 00:18:30.150 --> 00:18:33.049 it with attention. That is wild. It even conquered 00:18:33.049 --> 00:18:36.309 complex biology. AlphaFold used an attention 00:18:36.309 --> 00:18:38.670 -based architecture to solve the 50 -year -old 00:18:38.670 --> 00:18:40.950 grand challenge of protein folding. Oh, wow. 00:18:41.089 --> 00:18:43.910 Biology, too. Yeah. It treated individual amino 00:18:43.910 --> 00:18:47.150 acids as tokens in a sequence, and the attention 00:18:47.150 --> 00:18:49.309 mechanism figured out how they interact to predict 00:18:49.309 --> 00:18:52.009 a complex 3D biological structure. You know, 00:18:52.009 --> 00:18:53.890 I want to point out an absolutely mind -blowing 00:18:53.890 --> 00:18:55.990 fact from the source material regarding just 00:18:55.990 --> 00:18:58.490 how deeply this architecture understands relationships. 00:18:58.730 --> 00:19:01.349 The chess thing. Yes. A transformer learned to 00:19:01.349 --> 00:19:04.519 play chess at a grand master level. an ELO of 00:19:04.519 --> 00:19:07.640 2895. And it did this purely through static board 00:19:07.640 --> 00:19:10.019 evaluation. Which breaks every single rule of 00:19:10.019 --> 00:19:12.160 traditional chess engines. Right. Programs like 00:19:12.160 --> 00:19:14.880 Stockfish rely on massive search trees. They 00:19:14.880 --> 00:19:17.900 evaluate a board by calculating millions of future 00:19:17.900 --> 00:19:20.859 possibilities, branching out move after move. 00:19:21.109 --> 00:19:23.630 But this transformer didn't search ahead at all. 00:19:23.769 --> 00:19:26.089 It didn't look into the future. No. It took the 00:19:26.089 --> 00:19:28.930 64 squares and the pieces on them, converted 00:19:28.930 --> 00:19:32.130 that state into tokens, and processed them through 00:19:32.130 --> 00:19:35.569 its attention mechanisms. It just looked at the 00:19:35.569 --> 00:19:38.549 current board, understood the complex relationships 00:19:38.549 --> 00:19:41.529 and tensions between the pieces, and instantly 00:19:41.529 --> 00:19:43.930 knew the best move. It's credible. It understood 00:19:43.930 --> 00:19:46.369 the geometry of the board so deeply that it didn't 00:19:46.369 --> 00:19:48.829 need to calculate the future. It just recognized 00:19:48.829 --> 00:19:51.720 the pattern. It recognized the global dependencies 00:19:51.720 --> 00:19:55.059 of a chessboard the exact same way it recognizes 00:19:55.059 --> 00:19:58.160 the grammar of an English sentence. It really 00:19:58.160 --> 00:20:01.140 is a stunning testament to how universally powerful 00:20:01.140 --> 00:20:03.680 the attention mechanism really is. So what does 00:20:03.680 --> 00:20:06.950 this all actually mean? If we zoom out. The transformer 00:20:06.950 --> 00:20:09.690 changed the world by completely abandoning the 00:20:09.690 --> 00:20:11.990 human way of reading sequentially. It embraced 00:20:11.990 --> 00:20:15.609 massive parallel processing using queries, keys, 00:20:15.730 --> 00:20:19.309 and values to calculate attention, allowing AI 00:20:19.309 --> 00:20:22.650 to hold and understand deep context, not just 00:20:22.650 --> 00:20:25.390 in a paragraph of text, but in a grid of pixels, 00:20:25.750 --> 00:20:28.710 a wave of audio, or a chain of biology. So the 00:20:28.710 --> 00:20:31.329 next time you type a prompt into an AI, you can 00:20:31.329 --> 00:20:33.789 visualize the sheer scale of what is happening 00:20:33.789 --> 00:20:37.289 under the hood. before he answers you, there 00:20:37.289 --> 00:20:40.109 is a massive mathematical cocktail party happening 00:20:40.109 --> 00:20:43.470 inside the GPU. Thousands of queries and keys 00:20:43.470 --> 00:20:45.670 are mingling, measuring their similarity, pulling 00:20:45.670 --> 00:20:48.470 values, and passing through multiple multi -head 00:20:48.470 --> 00:20:51.109 lenses to understand the exact context of what 00:20:51.109 --> 00:20:53.170 you just asked. It really is beautiful when you 00:20:53.170 --> 00:20:54.970 break it down into the mechanics. Yeah. But I 00:20:54.970 --> 00:20:56.809 want to leave you with a final somewhat provocative 00:20:56.809 --> 00:20:59.890 thought to mull over. OK. If the exact same transformer 00:20:59.890 --> 00:21:02.420 architecture just dot products, matrices, and 00:21:02.420 --> 00:21:04.640 attention weights. If it can learn the grammar 00:21:04.640 --> 00:21:07.279 of human language and the grammar of pixels in 00:21:07.279 --> 00:21:10.119 a video, the grammar of amino acids in a biological 00:21:10.119 --> 00:21:12.980 protein, it raises an important question. Is 00:21:12.980 --> 00:21:15.339 everything in our universe ultimately just a 00:21:15.339 --> 00:21:18.180 sequence of tokens? And if so, what other secrets 00:21:18.180 --> 00:21:20.400 of the universe are just waiting for the right 00:21:20.400 --> 00:21:23.460 attention mechanism to decode them? That is a 00:21:23.460 --> 00:21:25.759 fascinating thought to take with you. Thank you 00:21:25.759 --> 00:21:28.400 so much for taking this deep dive with us. Keep 00:21:28.400 --> 00:21:30.809 questioning, keep learning. and we'll catch you 00:21:30.809 --> 00:21:31.190 next time.