WEBVTT 00:00:00.000 --> 00:00:02.779 What if I told you that the speed limit printed 00:00:02.779 --> 00:00:06.320 on your new computer's box is, well, it's essentially 00:00:06.320 --> 00:00:08.730 a multi -billion dollar fiction. Yeah. I mean, 00:00:08.810 --> 00:00:12.070 it really is a massive, highly lucrative illusion. 00:00:12.369 --> 00:00:15.390 Right. Because for decades, the biggest tech 00:00:15.390 --> 00:00:18.149 companies on earth have been locked in this, 00:00:18.149 --> 00:00:21.350 like, high stakes arms race to basically win 00:00:21.350 --> 00:00:24.230 at performance tests. Oh, absolutely. And behind 00:00:24.230 --> 00:00:26.910 the scenes, they are engineering deceptive hardware. 00:00:27.230 --> 00:00:29.089 They're rigging the very structure of the tests. 00:00:29.589 --> 00:00:32.399 And sometimes, well, sometimes... completely 00:00:32.399 --> 00:00:34.960 ignoring the laws of physics just to look good 00:00:34.960 --> 00:00:38.000 on a spreadsheet. It's crazy. So welcome to today's 00:00:38.000 --> 00:00:40.880 deep dive. We are exploring a really fascinating 00:00:40.880 --> 00:00:43.359 compilation of research today, anchored by the 00:00:43.359 --> 00:00:46.280 Wikipedia archives on Benchmark, specifically 00:00:46.280 --> 00:00:48.219 in computing. And, you know, our mission today 00:00:48.219 --> 00:00:50.539 isn't just to throw heavy technical definitions 00:00:50.539 --> 00:00:52.840 at you. No, definitely not. We're going to rip 00:00:52.840 --> 00:00:54.939 the lid off this whole world of hardware marketing. 00:00:55.340 --> 00:00:57.880 We want to explore the intense engineering tradeoffs 00:00:57.880 --> 00:01:00.500 happening in the dark and uncover the absolute 00:01:00.500 --> 00:01:02.619 lengths these companies go to in order to win 00:01:02.619 --> 00:01:04.700 the numbers game. Yeah, it gets pretty wild. 00:01:05.000 --> 00:01:07.200 It really does. So let's start with a baseline 00:01:07.200 --> 00:01:11.019 definition. A benchmark is fundamentally the 00:01:11.019 --> 00:01:14.510 act of running a specific computer program. or 00:01:14.510 --> 00:01:17.930 a set of operations to assess the relative performance 00:01:17.930 --> 00:01:20.810 of an object. Right. You basically run a standard 00:01:20.810 --> 00:01:23.069 trial against a machine to see how it does. Okay, 00:01:23.090 --> 00:01:26.250 let's unpack this. Because on the surface, I 00:01:26.250 --> 00:01:28.689 mean, that just sounds like basic quality control, 00:01:28.810 --> 00:01:30.530 right? Sure. It sounds perfectly reasonable. 00:01:30.849 --> 00:01:33.750 But why go through the trouble of designing these 00:01:33.750 --> 00:01:36.989 massive, elaborate obstacle courses? Like, why 00:01:36.989 --> 00:01:40.250 can't we just look at the raw physical specifications 00:01:40.250 --> 00:01:42.730 of the microchip. Well, because raw physical 00:01:42.730 --> 00:01:45.109 specifications can be incredibly misleading. 00:01:45.370 --> 00:01:47.469 I mean, historically, relying on those specs 00:01:47.469 --> 00:01:49.870 created a fundamental flaw in how computers were 00:01:49.870 --> 00:01:52.090 actually sold. Right. It led to something called 00:01:52.090 --> 00:01:54.370 the megahertz myth. The megahertz myth. OK, so 00:01:54.370 --> 00:01:56.129 this is about clock speed, right? Precisely. 00:01:56.469 --> 00:01:59.530 In the late 90s and early 2000s, computer architecture 00:01:59.530 --> 00:02:02.409 was, you know, evolving rapidly. But marketers 00:02:02.409 --> 00:02:04.709 were still selling machines based almost purely 00:02:04.709 --> 00:02:07.469 on clock frequency. Like megahertz and gigahertz. 00:02:07.709 --> 00:02:09.870 Exactly. And the classic example from the source 00:02:09.870 --> 00:02:13.009 is the Intel Pentium 4 processor. It generally 00:02:13.009 --> 00:02:15.870 operated at a dramatically higher clock frequency 00:02:15.870 --> 00:02:19.349 than its competitors at the time, like the Athlon 00:02:19.349 --> 00:02:23.430 XP or the PowerPC chips. So to a buyer standing 00:02:23.430 --> 00:02:26.830 in a store, looking at the boxes, the Pentium 00:02:26.830 --> 00:02:29.590 4 had the biggest number on the placard. It must 00:02:29.590 --> 00:02:32.110 be the fastest. Right. But a faster clock speed 00:02:32.110 --> 00:02:34.909 does not necessarily translate to more computational 00:02:34.909 --> 00:02:37.870 power. Wait, really? Why not? Because it all 00:02:37.870 --> 00:02:40.550 depends on how much actual work the processor 00:02:40.550 --> 00:02:43.349 gets done during each one of those individual 00:02:43.349 --> 00:02:45.949 clock cycles. Oh, I see. Think of a clock cycle 00:02:45.949 --> 00:02:47.849 like a worker hitting an anvil with a hammer. 00:02:48.610 --> 00:02:51.289 One worker might swing the hammer extremely fast, 00:02:51.349 --> 00:02:53.590 that's your high megahertz, but they are using 00:02:53.590 --> 00:02:56.229 a tiny hammer and barely venting the metal. Okay, 00:02:56.370 --> 00:02:58.169 yeah. Another worker swings half as fast, but 00:02:58.169 --> 00:03:00.650 they are using a massive sledgehammer and getting 00:03:00.650 --> 00:03:03.620 like... twice as much shaping done per swing. 00:03:03.740 --> 00:03:05.860 That's a great way to put it. It perfectly illustrates 00:03:05.860 --> 00:03:08.219 why comparing spec sheets is essentially like 00:03:08.219 --> 00:03:10.159 looking at a car's speedometer in a showroom. 00:03:10.439 --> 00:03:12.780 Oh, yeah, exactly. Just because the manufacturer 00:03:12.780 --> 00:03:15.300 printed 200 miles per hour on the dashboard, 00:03:16.000 --> 00:03:17.460 I mean, that doesn't mean the car can handle 00:03:17.460 --> 00:03:19.719 a tight corner. And it certainly doesn't mean 00:03:19.719 --> 00:03:22.039 it can haul a heavy trailer up a mountain. No, 00:03:22.099 --> 00:03:24.039 not at all. You can't just trust the dashboard. 00:03:24.439 --> 00:03:26.379 You have to put the car on a physical frack and 00:03:26.379 --> 00:03:28.099 see what happens when the rubber hits the road. 00:03:28.240 --> 00:03:30.840 And what's fascinating here is how the realization 00:03:30.840 --> 00:03:33.659 of that fact shifted the entire balance of power 00:03:33.659 --> 00:03:36.560 in the tech industry. How so? Well, early attempts 00:03:36.560 --> 00:03:38.719 to measure this real world speed were pretty 00:03:38.719 --> 00:03:42.120 rudimentary. For instance, Linux systems used 00:03:42.120 --> 00:03:45.699 a metric called BOGO MIPS. Wait, BOGO? Like bogus? 00:03:45.860 --> 00:03:48.719 Yeah, literally standing for bogus millions of 00:03:48.719 --> 00:03:50.879 instructions per second. That's hilarious. It 00:03:50.879 --> 00:03:52.900 was basically just a quick calibration loop the 00:03:52.900 --> 00:03:55.599 system ran during boot up. It didn't do any real 00:03:55.599 --> 00:03:57.719 work. It just measured how fast the processor 00:03:57.719 --> 00:03:59.879 could do absolutely nothing just to establish 00:03:59.879 --> 00:04:02.560 a baseline timing loop. Wow. OK. But as chips 00:04:02.560 --> 00:04:06.060 grew more complex, we needed real tracks. Benchmarks 00:04:06.060 --> 00:04:08.360 took power away from marketing departments who 00:04:08.360 --> 00:04:10.840 just wanted to sell the fastest hammer swing 00:04:10.840 --> 00:04:13.719 and handed it back to the engineers. And those 00:04:13.719 --> 00:04:16.610 engineers I mean, they aren't just using benchmarks 00:04:16.610 --> 00:04:19.430 to prove the marketers wrong. They use them to 00:04:19.430 --> 00:04:22.230 actually invent the future. Oh, absolutely. There's 00:04:22.230 --> 00:04:24.370 a section in our research about how processor 00:04:24.370 --> 00:04:27.550 architects use benchmarks internally before a 00:04:27.550 --> 00:04:30.879 chip even exists physically. which blew my mind. 00:04:31.000 --> 00:04:33.100 Yeah, this is one of the most critical applications. 00:04:33.300 --> 00:04:35.560 I mean, building a new microchip costs billions 00:04:35.560 --> 00:04:37.680 of dollars. You can't just manufacture one and 00:04:37.680 --> 00:04:39.600 hope it's fast. Right, that would be an expensive 00:04:39.600 --> 00:04:42.740 mistake. Exactly. So engineers take a benchmark, 00:04:42.920 --> 00:04:45.680 which is just a program that perfectly extracts 00:04:45.680 --> 00:04:48.300 the most intense performance -sensitive algorithms 00:04:48.300 --> 00:04:51.899 of a piece of software. Okay. And then they run 00:04:51.899 --> 00:04:54.480 that tiny snippet of code on what is called a 00:04:54.480 --> 00:04:57.259 cycle -accurate simulator. Wait, wait. So they're 00:04:57.259 --> 00:04:59.709 running a simulation of a computer inside another 00:04:59.709 --> 00:05:02.949 computer? Essentially, yes. It is a highly complex 00:05:02.949 --> 00:05:05.829 virtual sandbox that perfectly mimics the behavior 00:05:05.829 --> 00:05:09.569 of the unbuilt silicon down to the exact nanosecond 00:05:09.569 --> 00:05:12.610 of every single clock cycle. That is insane. 00:05:12.810 --> 00:05:15.810 It's incredible. It allows engineers to see exactly 00:05:15.810 --> 00:05:18.149 where the bottlenecks are, giving them precise 00:05:18.149 --> 00:05:21.389 clues on how to physically rearrange the microscopic 00:05:21.389 --> 00:05:24.189 pathways of the chip to improve performance, 00:05:24.430 --> 00:05:26.970 long before anyone fires up a silicon forge. 00:05:27.310 --> 00:05:29.730 OK, so if we agree that we have to put the car 00:05:29.730 --> 00:05:32.170 on the track to see what it can actually do, 00:05:32.689 --> 00:05:34.769 we need to look at the tracks themselves. Right, 00:05:34.850 --> 00:05:37.290 the benchmarks. Because engineers have designed 00:05:37.290 --> 00:05:40.449 a massive arsenal of different tests, and they 00:05:40.449 --> 00:05:43.980 are definitely not all created equal. Broadly 00:05:43.980 --> 00:05:46.500 speaking, there are application benchmarks and 00:05:46.500 --> 00:05:49.279 synthetic benchmarks. Yeah. Application benchmarks 00:05:49.279 --> 00:05:51.560 are the most straightforward. You just take real 00:05:51.560 --> 00:05:54.339 -world programs and time how long a system takes 00:05:54.339 --> 00:05:56.220 to execute them. Like what kind of programs? 00:05:56.439 --> 00:05:58.639 Well, you might measure how fast a machine can 00:05:58.639 --> 00:06:00.959 compile the millions of lines of code required 00:06:00.959 --> 00:06:03.459 to build the Chromium web browser from scratch. 00:06:03.540 --> 00:06:06.300 OK, yeah. Or more commonly, for regular consumers, 00:06:06.480 --> 00:06:08.500 you just run high -end, graphically demanding 00:06:08.500 --> 00:06:11.100 video games and measure the frame rate. That 00:06:11.100 --> 00:06:13.300 makes intuitive sense because it represents exactly 00:06:13.300 --> 00:06:14.920 what the user is actually going to do with the 00:06:14.920 --> 00:06:16.779 machine. But then we have synthetic benchmarks, 00:06:17.040 --> 00:06:19.240 like the classic wet stone or dry stone tests. 00:06:20.139 --> 00:06:22.040 And the way these are built seems, I don't know, 00:06:22.160 --> 00:06:24.370 almost entirely disconnected from reality. It 00:06:24.370 --> 00:06:26.629 does seem that way at first glance. Because programmers 00:06:26.629 --> 00:06:29.689 do a statistical analysis of the types of operations 00:06:29.689 --> 00:06:31.730 used across dozens of different applications. 00:06:32.290 --> 00:06:33.889 They figure out the mathematical proportions, 00:06:34.310 --> 00:06:37.170 say 20 % addition, 10 % moving memory around, 00:06:37.529 --> 00:06:40.689 5 % complex logic. And then they write a completely 00:06:40.689 --> 00:06:44.069 artificial Frankenstein program based on those 00:06:44.069 --> 00:06:46.720 exact proportions. Right. Why would anyone bother 00:06:46.720 --> 00:06:49.319 using a mathematical mimic if they could just 00:06:49.319 --> 00:06:51.279 run the real thing? I mean, isn't a synthetic 00:06:51.279 --> 00:06:53.439 benchmark a step backward? It looks that way 00:06:53.439 --> 00:06:56.139 on the surface, sure, but the driving force here 00:06:56.139 --> 00:06:59.189 is isolation. Isolation. Yeah. Think about a 00:06:59.189 --> 00:07:01.610 massive real -world application like running 00:07:01.610 --> 00:07:04.769 a heavy video editor. That software relies on 00:07:04.769 --> 00:07:07.610 the CPU, the system memory, the storage drive, 00:07:07.750 --> 00:07:10.009 the graphics card, all of it simultaneously. 00:07:10.189 --> 00:07:11.910 All right, there's a lot going on. So if you 00:07:11.910 --> 00:07:14.550 are an engineer trying to test the exact latency 00:07:14.550 --> 00:07:17.589 of a brand new experimental hard disk, a real 00:07:17.589 --> 00:07:20.129 -world application is a nightmare. The waters 00:07:20.129 --> 00:07:22.569 are too muddy. Oh, I see. If the video editor 00:07:22.569 --> 00:07:25.110 stutters, you have no idea if your new hard disk 00:07:25.110 --> 00:07:28.550 caused it or if the system memory got... Overloaded 00:07:28.550 --> 00:07:30.949 or if the video editor just has terribly written 00:07:30.949 --> 00:07:34.540 code So the synthetic test is basically a sterilized 00:07:34.540 --> 00:07:36.680 laboratory environment. Exactly. You strip away 00:07:36.680 --> 00:07:38.600 all the other variables. This is also why we 00:07:38.600 --> 00:07:41.740 rely so heavily on microbenchmarks. Right. These 00:07:41.740 --> 00:07:44.819 are tiny, hyper -specific pieces of code designed 00:07:44.819 --> 00:07:47.740 to stress test one single hardware component 00:07:47.740 --> 00:07:50.240 in a complete vacuum. So just one piece at a 00:07:50.240 --> 00:07:52.579 time. Right. If you want to test a network switch, 00:07:52.819 --> 00:07:55.480 you don't load up a web page. You blast it with 00:07:55.480 --> 00:07:58.480 a synthetic microbenchmark that sends specific 00:07:58.480 --> 00:08:02.420 raw packets of data just to see at what millisecond 00:08:02.420 --> 00:08:04.920 the switch drops a packet. That level of isolation 00:08:04.920 --> 00:08:07.879 explains some of the wild acronyms we see in 00:08:07.879 --> 00:08:11.220 this space like LINPACK. Oh LINPACK, yes. It's 00:08:11.220 --> 00:08:13.180 an open -source standard historically used to 00:08:13.180 --> 00:08:15.980 measure FLOPs and let's actually pause on FLOPs 00:08:15.980 --> 00:08:17.399 for a second because it's thrown around a lot 00:08:17.399 --> 00:08:20.560 with supercomputers. What is a FLOPCAC mechanically? 00:08:20.980 --> 00:08:23.819 So FLOPs stands for floating point operations 00:08:23.819 --> 00:08:27.540 per second. Mechanically, it's a measure of the 00:08:27.540 --> 00:08:30.199 computer's ability to handle extremely complex 00:08:30.199 --> 00:08:33.360 numbers, specifically numbers with decimal points 00:08:33.360 --> 00:08:36.500 that can float or move around to represent very 00:08:36.500 --> 00:08:39.679 large or very small quantities. Basic arithmetic, 00:08:39.879 --> 00:08:43.159 like adding 2 plus 2, is an integer operation. 00:08:43.240 --> 00:08:46.120 It's easy for a computer. But calculating the 00:08:46.120 --> 00:08:49.460 precise trajectory of a weather system, or rendering 00:08:49.460 --> 00:08:52.139 the 3D physics of light bouncing off a wet road 00:08:52.139 --> 00:08:54.799 in a video game, that requires massive amounts 00:08:54.799 --> 00:08:58.059 of floating point math. Wow, OK. So LINPACK essentially 00:08:58.059 --> 00:09:00.759 throws a wall of these complex linear algebra 00:09:00.759 --> 00:09:03.700 equations at a system to see exactly how many 00:09:03.700 --> 00:09:05.539 it can solve per second. That makes a lot of 00:09:05.539 --> 00:09:07.720 sense. Yeah. And as the technology gets weirder, 00:09:07.909 --> 00:09:10.230 Tests have to get weirder, too. Like, my absolute 00:09:10.230 --> 00:09:12.389 favorite detail from the research is the Will 00:09:12.389 --> 00:09:15.450 Smith eating spaghetti test. Oh, yeah. A modern 00:09:15.450 --> 00:09:17.409 classic in the artificial intelligence space. 00:09:17.629 --> 00:09:19.509 It sounds like a complete joke, but it's used 00:09:19.509 --> 00:09:22.450 as an informal benchmark for new text -to -video 00:09:22.450 --> 00:09:25.169 AI models. It really is. And when you think about 00:09:25.169 --> 00:09:28.629 it, it perfectly illustrates the problem of testing 00:09:28.629 --> 00:09:32.610 cutting -edge tech. How do you grade an AI's 00:09:32.610 --> 00:09:34.559 imagination? Right, you can't just ask it to 00:09:34.559 --> 00:09:36.659 solve a math problem. Exactly. You have to ask 00:09:36.659 --> 00:09:39.700 it to render something completely unnatural and 00:09:39.700 --> 00:09:43.659 bizarre like Will Smith aggressively eating spaghetti 00:09:43.659 --> 00:09:47.159 to see if the AI understands the physics of noodles, 00:09:47.940 --> 00:09:51.500 the mechanics of human anatomy, and object permanence. 00:09:51.639 --> 00:09:54.919 Or if it just hallucinates a nightmarish morphing 00:09:54.919 --> 00:09:57.740 blob of pasta. Which we've all seen. And it is 00:09:57.740 --> 00:10:00.289 terrifying. It really is a great example of how 00:10:00.289 --> 00:10:02.409 benchmarks have to evolve to meet the technology. 00:10:02.769 --> 00:10:05.470 But, you know, there is a very dark side to this 00:10:05.470 --> 00:10:07.230 evolution. Yeah, there is. Because the moment 00:10:07.230 --> 00:10:08.870 you establish a track, the competitors don't 00:10:08.870 --> 00:10:10.730 just figure out how to drive perfectly on that 00:10:10.730 --> 00:10:13.070 track. They figure out how to cheat. Yes. Here's 00:10:13.070 --> 00:10:15.509 where it gets really interesting. The benchmark 00:10:15.509 --> 00:10:19.070 wars. Because once a specific test becomes the 00:10:19.070 --> 00:10:21.950 industry standard, millions of dollars in sales 00:10:21.950 --> 00:10:24.690 hinge on winning it. Tens of millions, easily. 00:10:24.960 --> 00:10:27.779 We saw this heavily in the 1980s and 1990s with 00:10:27.779 --> 00:10:30.899 the massive relational database makers. They 00:10:30.899 --> 00:10:33.320 would plaster benchmark scores all over their 00:10:33.320 --> 00:10:35.559 marketing, but the numbers were effectively rigged. 00:10:35.700 --> 00:10:38.080 They were meticulously manipulated. And it's 00:10:38.080 --> 00:10:41.279 a fully documented tactic. Vendors actively tune 00:10:41.279 --> 00:10:44.440 their systems to ace specific standard tests, 00:10:44.940 --> 00:10:47.039 entirely ignoring how the machine will function 00:10:47.039 --> 00:10:49.299 in the real world. Right. Take a historically 00:10:49.299 --> 00:10:52.419 prominent test like Norton's SysInfo. It was 00:10:52.419 --> 00:10:54.580 heavily biased toward measuring the speed of 00:10:54.580 --> 00:10:57.100 multiple concurrent operations. Boy, how does 00:10:57.100 --> 00:11:00.000 a machine actually memorize a test? Like if I 00:11:00.000 --> 00:11:01.879 take a standardized test, I can memorize the 00:11:01.879 --> 00:11:04.419 answers. How does a microchip do it? It often 00:11:04.419 --> 00:11:06.259 happens at the compiler level. The compiler? 00:11:06.399 --> 00:11:08.580 Yeah, the compiler is the translator that turns 00:11:08.580 --> 00:11:11.059 human written software code into the binary language 00:11:11.059 --> 00:11:13.500 the hardware actually understands. OK. Hardware 00:11:13.500 --> 00:11:15.899 vendors would literally write specific, hidden 00:11:15.899 --> 00:11:18.700 rules into their compilers. The compiler would 00:11:18.700 --> 00:11:21.519 be trained to recognize the exact sequence of 00:11:21.519 --> 00:11:24.019 math problems that Norton Sysinfo or another 00:11:24.019 --> 00:11:27.299 benchmark was about to ask. Oh. So when it spotted 00:11:27.299 --> 00:11:30.419 that specific test, the compiler would say, Don't 00:11:30.419 --> 00:11:32.419 actually do the hard computational work. I already 00:11:32.419 --> 00:11:35.039 know the answer. Just swap in this pre -calculated 00:11:35.039 --> 00:11:38.360 shortcut. That is wild. It's the literal equivalent 00:11:38.360 --> 00:11:42.259 of a student stealing the answer key to the SATs. 00:11:42.580 --> 00:11:44.139 That's exactly what it is. They get a perfect 00:11:44.139 --> 00:11:45.879 score, but they haven't actually learned any 00:11:45.879 --> 00:11:48.159 math. So when they graduate and go to do the 00:11:48.159 --> 00:11:51.679 actual job, they are completely useless. Exactly. 00:11:52.179 --> 00:11:54.340 And this raises an important question about the 00:11:54.340 --> 00:11:57.440 foundation of trust in tech marketing. If a vendor 00:11:57.440 --> 00:12:00.799 is willing to write bypass code, just to inflate 00:12:00.799 --> 00:12:03.399 a marketing metric, what else are they cutting 00:12:03.399 --> 00:12:06.039 corners on? Seriously. And it isn't just silicon 00:12:06.039 --> 00:12:08.700 tweaks either. They also manipulate the financial 00:12:08.700 --> 00:12:12.519 metrics. This brings us to the infamous benchmark 00:12:12.519 --> 00:12:14.720 special. Right, because enterprise buyers aren't 00:12:14.720 --> 00:12:16.740 just looking at speed. They're looking at total 00:12:16.740 --> 00:12:19.820 cost of ownership, or PCO. They want the best 00:12:19.820 --> 00:12:22.559 performance for the lowest price. Exactly. So 00:12:22.559 --> 00:12:25.259 to win the TCO metric on a benchmark, vendors 00:12:25.259 --> 00:12:28.159 design a benchmark special. Which is what exactly? 00:12:28.330 --> 00:12:32.169 This is a highly specific mutant hardware configuration 00:12:32.169 --> 00:12:35.029 that has an artificially low price tag. They 00:12:35.029 --> 00:12:37.070 completely strip out anything that isn't strictly 00:12:37.070 --> 00:12:39.649 required to pass the test. But wait, if I'm running 00:12:39.649 --> 00:12:42.269 a major corporate data center, I can't run a 00:12:42.269 --> 00:12:44.610 stripped down mutant server. I need disaster 00:12:44.610 --> 00:12:46.870 recovery. I need background environments for 00:12:46.870 --> 00:12:49.649 my developers to test new software without crashing 00:12:49.649 --> 00:12:51.970 the main system. Right, and the benchmark special 00:12:51.970 --> 00:12:55.059 includes absolutely none of that. It only reports 00:12:55.059 --> 00:12:57.360 the computing capacity needed to cross the finish 00:12:57.360 --> 00:13:00.259 line of the benchmark itself. So it's completely 00:13:00.259 --> 00:13:03.480 unrealistic? Entirely. If a real business bought 00:13:03.480 --> 00:13:05.919 that exact configuration and then tried to add 00:13:05.919 --> 00:13:08.059 in the vital safety nets, the backups, and the 00:13:08.059 --> 00:13:10.700 background tasks, the actual cost of the system 00:13:10.700 --> 00:13:13.440 would skyrocket. The benchmark TCO is a complete 00:13:13.440 --> 00:13:16.169 fantasy. It's stunning how poor the scientific 00:13:16.169 --> 00:13:18.590 method can be in this space. I mean, you have 00:13:18.590 --> 00:13:21.230 tests with small sample sizes, zero variable 00:13:21.230 --> 00:13:23.830 control, and results that independent labs can't 00:13:23.830 --> 00:13:26.470 even replicate. But deliberate cheating is really 00:13:26.470 --> 00:13:29.710 only half the problem. Even the honest benchmarks 00:13:29.710 --> 00:13:32.769 suffer from massive blind spots. There are real 00:13:32.769 --> 00:13:35.230 world realities that standard tests just fail 00:13:35.230 --> 00:13:38.169 to capture. Let's look at the first major blind 00:13:38.169 --> 00:13:42.049 spot from the source, the 100 % cliff. Ah, yes. 00:13:42.049 --> 00:13:44.570 This is a classic trap. When vendors publish 00:13:44.570 --> 00:13:46.909 server benchmarks, they usually run the tests 00:13:46.909 --> 00:13:49.990 at a continuous usage of about 80%. Which feels 00:13:49.990 --> 00:13:52.370 like a heavy load. I mean, if my car runs great 00:13:52.370 --> 00:13:55.289 at 80 % of its maximum RPMs for hours, I'm pretty 00:13:55.289 --> 00:13:58.210 happy. It is a heavy load, sure, but it is a 00:13:58.210 --> 00:14:00.169 perfectly controlled steady state. Right. The 00:14:00.169 --> 00:14:02.110 real world doesn't operate in a steady state. 00:14:02.250 --> 00:14:04.990 It operates in sudden chaotic spikes. And the 00:14:04.990 --> 00:14:07.250 mechanical reality is that many server architectures 00:14:07.250 --> 00:14:09.470 degrade catastrophically when demand spikes from 00:14:09.470 --> 00:14:12.409 80 % to 100%. They essentially fall off a cliff. 00:14:12.529 --> 00:14:14.490 But why does the cliff happen mechanically? Like, 00:14:14.509 --> 00:14:16.730 why doesn't it just run 20 % slower? Because 00:14:16.730 --> 00:14:19.570 it runs out of its fastest resources. At 80%, 00:14:19.570 --> 00:14:21.769 the processor is comfortably juggling data in 00:14:21.769 --> 00:14:24.250 its ultra -fast short -term memory. So the cache 00:14:24.250 --> 00:14:27.370 and the RAM. OK, that makes sense. But at 100%, 00:14:27.370 --> 00:14:30.419 that fast memory fills up entirely. Suddenly, 00:14:30.580 --> 00:14:33.259 the system has to start dumping active data down 00:14:33.259 --> 00:14:35.679 into the storage drive, which is exponentially 00:14:35.679 --> 00:14:38.840 slower. It creates a massive system -wide traffic 00:14:38.840 --> 00:14:42.639 jam. So a benchmark running at a steady 80 % 00:14:42.639 --> 00:14:45.779 will completely fail to document the catastrophic 00:14:45.779 --> 00:14:48.639 lockup that happens when a surge of real traffic 00:14:48.639 --> 00:14:50.259 hits. You know, you don't have to be a server 00:14:50.259 --> 00:14:52.259 administrator to understand this. Anyone with 00:14:52.259 --> 00:14:54.679 a smartphone knows this pain perfectly. Oh, definitely. 00:14:54.759 --> 00:14:56.820 You don't care about the mathematically average 00:14:56.820 --> 00:14:59.049 speed of your phone over a 30 -day period. period, 00:14:59.470 --> 00:15:01.929 you care that the device completely locked up 00:15:01.929 --> 00:15:04.350 and froze for 15 seconds right when you were 00:15:04.350 --> 00:15:06.070 at the front of the airport security line trying 00:15:06.070 --> 00:15:08.129 to pull up your boarding pass. That perfectly 00:15:08.129 --> 00:15:10.909 captures the disconnect between IT perception 00:15:10.909 --> 00:15:14.490 and user perception. Benchmarks emphasize mean 00:15:14.490 --> 00:15:17.230 scores averages. They like to report that the 00:15:17.230 --> 00:15:19.549 system responds in 50 milliseconds on average. 00:15:20.090 --> 00:15:22.789 But users don't feel averages. They feel anomalies. 00:15:22.889 --> 00:15:25.919 We only notice when it breaks. Exactly. Users 00:15:25.919 --> 00:15:28.820 want predictability. They want a low standard 00:15:28.820 --> 00:15:31.480 deviation, meaning the speed is incredibly consistent. 00:15:32.000 --> 00:15:35.299 A system that is lightning fast 99 % of the time, 00:15:35.360 --> 00:15:38.200 but completely freezes for a full minute 1 % 00:15:38.200 --> 00:15:41.340 of the time. Well, it might yield a fantastic 00:15:41.340 --> 00:15:44.179 average benchmark score, but it provides an absolutely 00:15:44.179 --> 00:15:46.500 miserable user experience. And there's another 00:15:46.500 --> 00:15:49.059 massive blind spot that seems so obvious once 00:15:49.059 --> 00:15:51.299 you really think about it. The electricity bill. 00:15:51.659 --> 00:15:55.039 The physical facility's burden. Yes. Benchmarks 00:15:55.039 --> 00:15:57.179 are generally obsessed with the speed of computation, 00:15:57.320 --> 00:15:59.899 but they operate as if the laws of physics just 00:15:59.899 --> 00:16:02.720 don't exist. Faster switching in semiconductors 00:16:02.720 --> 00:16:05.100 almost universally requires pulling more electrical 00:16:05.100 --> 00:16:07.559 power. Right. And more power brings an immediate 00:16:07.559 --> 00:16:09.799 cascade of real -world consequences. I mean, 00:16:09.879 --> 00:16:12.500 if you were buying laptops for a fleet of traveling 00:16:12.500 --> 00:16:16.220 salespeople, a chip that aces a speed benchmark 00:16:16.220 --> 00:16:19.179 but drains the battery in two hours is Basically 00:16:19.179 --> 00:16:21.519 useless. Completely useless. And on the enterprise 00:16:21.519 --> 00:16:23.679 scale, if you were building a data center, consuming 00:16:23.679 --> 00:16:26.019 more power means generating massive amounts of 00:16:26.019 --> 00:16:28.460 heat. Oh wow, yeah. A server rack might benchmark 00:16:28.460 --> 00:16:31.159 as the fastest in the world, but if it physically 00:16:31.159 --> 00:16:33.179 melts the cables because your building's cooling 00:16:33.179 --> 00:16:35.840 system cannot handle the thermal outpane, that 00:16:35.840 --> 00:16:39.159 speed is totally irrelevant. By ignoring performance 00:16:39.159 --> 00:16:42.740 per watt, space, and cooling, benchmarks ignore 00:16:42.740 --> 00:16:45.279 the physical reality of the hardware. It's crazy 00:16:45.279 --> 00:16:47.779 how much gets left out. And there's one more 00:16:47.779 --> 00:16:50.659 major blind spot we need to touch on. The batch 00:16:50.659 --> 00:16:53.559 problem. Right. Most modern benchmarks focus 00:16:53.559 --> 00:16:56.320 on interactive speed like how fast the system 00:16:56.320 --> 00:16:58.440 responds to a single user clicking a button. 00:16:59.100 --> 00:17:01.620 But the backbone of global commerce doesn't run 00:17:01.620 --> 00:17:05.400 on single clicks. It runs on massive concurrent 00:17:05.400 --> 00:17:08.420 batch processing. Think about a massive telecommunications 00:17:08.420 --> 00:17:10.720 company doing its end of month billing. They 00:17:10.720 --> 00:17:12.400 aren't worried about millisecond response times 00:17:12.400 --> 00:17:15.140 for one customer. They have to accurately crunch 00:17:15.140 --> 00:17:18.740 50 million account records before a strict 660 00:17:18.740 --> 00:17:21.660 re -zero AM deadline. Exactly. It is all about 00:17:21.660 --> 00:17:24.339 the absolute predictability of completing high 00:17:24.339 --> 00:17:27.359 -volume, long -running tasks concurrently. Standard 00:17:27.359 --> 00:17:29.299 benchmarks struggle immensely to measure this. 00:17:29.380 --> 00:17:31.559 Because it's too complex to simulate easily. 00:17:31.940 --> 00:17:34.259 Partially, yes. And if we connect this to the 00:17:34.259 --> 00:17:36.920 bigger picture, the entire paradigm of how we 00:17:36.920 --> 00:17:40.450 test is fundamentally outdated. Today, data centers 00:17:40.450 --> 00:17:43.630 rely heavily on virtualization and grid computing. 00:17:43.829 --> 00:17:45.890 Right, slicing up the servers. Yes, they take 00:17:45.890 --> 00:17:49.230 a massive physical server and slice it into... 00:17:49.019 --> 00:17:51.859 dozens of virtual servers running completely 00:17:51.859 --> 00:17:54.759 different application tiers simultaneously. Measuring 00:17:54.759 --> 00:17:57.579 how fast one isolated application runs on one 00:17:57.579 --> 00:18:00.480 isolated server tells you almost nothing about 00:18:00.480 --> 00:18:03.240 how a modern, consolidated data center will actually 00:18:03.240 --> 00:18:06.119 perform. So we have a landscape filled with rigged 00:18:06.119 --> 00:18:08.680 compilers, benchmark specials that lie about 00:18:08.680 --> 00:18:11.740 price, systems that fall off 100 % cliffs, and 00:18:11.740 --> 00:18:13.660 tests that completely ignore the fact that computers 00:18:13.660 --> 00:18:16.259 run on actual electricity. How does the industry 00:18:16.259 --> 00:18:19.509 bring any order to this chaos? Mostly by relying 00:18:19.509 --> 00:18:22.089 on independent watchdog organizations. You can 00:18:22.089 --> 00:18:24.190 think of groups like SPEC, the Standard Performance 00:18:24.190 --> 00:18:27.890 Evaluation Corporation, and the TPC, the Transaction 00:18:27.890 --> 00:18:30.329 Processing Performance Council, as basically 00:18:30.329 --> 00:18:33.049 the tech world's health inspectors. Health inspectors. 00:18:33.069 --> 00:18:35.269 I like that. Yeah. They enforce strict rules, 00:18:35.529 --> 00:18:37.990 these seven vital benchmarking principles like 00:18:37.990 --> 00:18:41.190 relevance, equity, repeat a builder to keep the 00:18:41.190 --> 00:18:43.970 tracks fair. But instead of just a dry list of 00:18:43.970 --> 00:18:46.920 textbook rules, What do these health inspectors 00:18:46.920 --> 00:18:49.339 actually demand from hardware makers in practice? 00:18:50.200 --> 00:18:52.500 Well, they demand transparency and real -world 00:18:52.500 --> 00:18:55.559 stakes. For instance, the test has to be repeatable 00:18:55.559 --> 00:18:58.240 by an independent third -party lab. You can't 00:18:58.240 --> 00:19:00.640 just publish numbers from your own secret testing 00:19:00.640 --> 00:19:02.680 facility. Right. No grading your own homework. 00:19:02.980 --> 00:19:05.759 Exactly. The tests have to be relatively affordable 00:19:05.759 --> 00:19:08.180 to run so smaller companies can actually compete. 00:19:08.400 --> 00:19:10.720 And the track cannot be secretly built to favor 00:19:10.720 --> 00:19:14.619 one specific brand's silicon architecture. Furthermore, 00:19:15.180 --> 00:19:17.420 organizations like the TPC combat the cheating 00:19:17.420 --> 00:19:20.119 we saw in the database wars by forcing companies 00:19:20.119 --> 00:19:23.059 to run ACID tests. Okay, let's clarify that because 00:19:23.059 --> 00:19:25.240 they aren't pouring battery asset on the servers. 00:19:25.680 --> 00:19:28.220 What is an ACID test mechanically? No, definitely 00:19:28.220 --> 00:19:30.640 not battery acid. ACID stands for Atomicity, 00:19:30.980 --> 00:19:34.440 Consistency, Isolation, Durability. It is a strict 00:19:34.440 --> 00:19:36.720 set of mechanical rules guaranteeing that data 00:19:36.720 --> 00:19:39.140 is safely handled even during a total system 00:19:39.140 --> 00:19:42.319 failure. Oh, OK, like what kind of failure? Well, 00:19:42.519 --> 00:19:45.160 it ensures that if a database crashes halfway 00:19:45.160 --> 00:19:47.920 through processing a $10 ,000 bank transfer, 00:19:48.420 --> 00:19:50.359 the money doesn't just vanish into thin air and 00:19:50.359 --> 00:19:52.420 it doesn't accidentally get deposited twice. 00:19:52.539 --> 00:19:55.039 Got it. It forces the system to do the heavy 00:19:55.039 --> 00:19:57.539 secure lifting that a real bank would require, 00:19:58.119 --> 00:20:00.700 making it impossible to use a stripped down benchmark 00:20:00.700 --> 00:20:03.779 special to just win the test. Okay, but even 00:20:03.779 --> 00:20:05.819 with these health inspectors enforcing the rules, 00:20:06.220 --> 00:20:09.019 the research delivers one ultimate undeniable 00:20:09.019 --> 00:20:11.960 truth about this entire industry. When performance 00:20:11.960 --> 00:20:14.519 is absolutely critical, the only benchmark that 00:20:14.519 --> 00:20:16.839 matters is the target environment's application 00:20:16.839 --> 00:20:18.960 suite. Period. That is it. So what does this 00:20:18.960 --> 00:20:21.539 all mean? It means that all the synthetic tests, 00:20:21.839 --> 00:20:24.819 all the fancy micro benchmarks, the LINPACK FLOP 00:20:24.819 --> 00:20:27.680 ES and the megahertz, they are just tools to 00:20:27.680 --> 00:20:29.859 make an educated guess. That's all they are. 00:20:30.059 --> 00:20:32.000 If you are a hospital buying a multi -million 00:20:32.000 --> 00:20:35.160 dollar server to run your specific patient record 00:20:35.160 --> 00:20:38.099 software, the only way to know exactly how fast 00:20:38.099 --> 00:20:40.500 it will run that software is to force the vendor 00:20:40.500 --> 00:20:42.559 to install your software on their server and 00:20:42.559 --> 00:20:45.859 test it yourself. That is the absolute only guarantee. 00:20:46.559 --> 00:20:48.700 Knowledge is only valuable when you apply it 00:20:48.700 --> 00:20:51.950 with critical thinking. You must look past the 00:20:51.950 --> 00:20:54.289 wall of numbers on the spec sheet and ask the 00:20:54.289 --> 00:20:57.250 fundamental question, does this metric actually 00:20:57.250 --> 00:21:00.049 represent the specific mechanical workload I 00:21:00.049 --> 00:21:02.589 need this machine to do? If it doesn't align 00:21:02.589 --> 00:21:05.009 with your reality, the number is completely irrelevant. 00:21:05.329 --> 00:21:08.230 It's a massive shift in how you evaluate technology. 00:21:08.700 --> 00:21:10.599 Well, thank you for taking this journey with 00:21:10.599 --> 00:21:13.339 us today. We've gone from the early days of the 00:21:13.339 --> 00:21:15.880 megahertz myth, unpacked the mechanics of the 00:21:15.880 --> 00:21:18.839 Will Smith spaghetti test, and exposed the compiler 00:21:18.839 --> 00:21:21.420 -rigging dark arts of the benchmark wars. It's 00:21:21.420 --> 00:21:24.079 been quite a ride. It has. Hopefully the next 00:21:24.079 --> 00:21:26.259 time you are staring at a massive wall of tech 00:21:26.259 --> 00:21:28.220 specs, you'll know exactly what questions to 00:21:28.220 --> 00:21:30.660 ask. It really is a relentlessly fascinating 00:21:30.660 --> 00:21:33.140 ecosystem. And, you know, before we wrap up, 00:21:33.220 --> 00:21:35.880 there was one... final concept from the research 00:21:35.880 --> 00:21:38.900 that points to where this is all heading. Oh. 00:21:39.180 --> 00:21:42.339 It's called continuous benchmarking. In modern 00:21:42.339 --> 00:21:44.440 software engineering, benchmarking isn't just 00:21:44.440 --> 00:21:47.099 a test you run once before a product launches 00:21:47.099 --> 00:21:50.440 anymore. It is being woven directly into the 00:21:50.440 --> 00:21:53.490 daily build pipelines. What does that mean exactly? 00:21:53.690 --> 00:21:56.309 It means every single day as developers write 00:21:56.309 --> 00:21:59.849 new code, the system automatically runs synthetic 00:21:59.849 --> 00:22:02.569 and application benchmarks against itself. Wow. 00:22:02.690 --> 00:22:04.650 So the software is constantly stress testing 00:22:04.650 --> 00:22:07.849 its own new layers. Precisely. And if we follow 00:22:07.849 --> 00:22:11.190 that trajectory into the age of artificial intelligence, 00:22:11.309 --> 00:22:14.269 we are moving toward a near future where computing 00:22:14.269 --> 00:22:17.150 systems constantly measure, evaluate, and perhaps 00:22:17.150 --> 00:22:19.509 eventually physically redesign their own pathways 00:22:19.509 --> 00:22:22.759 in real time based on their own internal testing. 00:22:23.099 --> 00:22:25.579 That's wild to think about. It raises a fascinating 00:22:25.579 --> 00:22:27.500 philosophical question for you to mull over. 00:22:27.799 --> 00:22:29.759 If our machines are continuously taking their 00:22:29.759 --> 00:22:32.500 own tests and writing their own tracks, who is 00:22:32.500 --> 00:22:35.079 really holding the grading pen? Wow. Something 00:22:35.079 --> 00:22:36.720 to really think about the next time you're staring 00:22:36.720 --> 00:22:39.660 at that wall of numbers on a box. Thanks for 00:22:39.660 --> 00:22:40.839 joining us on this deep dive.