1 00:00:00,000 --> 00:00:02,240 These models are going to get better. 2 00:00:02,240 --> 00:00:03,600 They're going to do more amazing things. 3 00:00:03,600 --> 00:00:06,720 It's an exciting time for us to be in. 4 00:00:06,720 --> 00:00:09,920 But as these models get generally better, 5 00:00:09,920 --> 00:00:13,260 this problem of like, all right, well, when it fails, 6 00:00:13,260 --> 00:00:16,120 knowing how it fails and doing everything we can 7 00:00:16,120 --> 00:00:17,960 to like inform the user and protect against it 8 00:00:17,960 --> 00:00:20,520 is going to become even bigger 9 00:00:20,520 --> 00:00:22,960 because we're going to start trusting these things more. 10 00:00:22,960 --> 00:00:28,460 How did the best machine learning practitioners 11 00:00:28,460 --> 00:00:30,440 get involved in the field? 12 00:00:30,440 --> 00:00:32,920 What challenges have they faced? 13 00:00:32,920 --> 00:00:35,080 What has helped them flourish? 14 00:00:35,080 --> 00:00:36,880 Let's ask them. 15 00:00:36,880 --> 00:00:39,320 Welcome to Learning from Machine Learning. 16 00:00:39,320 --> 00:00:41,480 I'm your host, Seth Levine. 17 00:00:41,480 --> 00:00:44,320 Hello and welcome to Learning from Machine Learning. 18 00:00:44,320 --> 00:00:47,080 On this episode, we have a very special guest, 19 00:00:47,080 --> 00:00:51,200 Chris Van Pelt, the co-founder of Weights and Biases, 20 00:00:51,200 --> 00:00:53,920 the co-founder of CrowdFlower and Figure 8, 21 00:00:53,920 --> 00:00:57,460 and somebody who's dedicated his career optimizing 22 00:00:57,460 --> 00:01:00,480 ML workflows and teaching ML practitioners, 23 00:01:00,480 --> 00:01:03,000 making machine learning more accessible to all. 24 00:01:03,000 --> 00:01:06,720 Chris, it is an absolute pleasure to have you on the show. 25 00:01:06,720 --> 00:01:09,400 It's a pleasure to be here. 26 00:01:09,400 --> 00:01:12,840 Why don't you start us off with what attracted you 27 00:01:12,840 --> 00:01:13,800 to machine learning? 28 00:01:16,680 --> 00:01:20,520 Yeah, this was quite a while ago. 29 00:01:20,520 --> 00:01:25,400 But I remember all the way back in college, 30 00:01:25,400 --> 00:01:30,480 studying computer science in the early 2000s, 31 00:01:30,480 --> 00:01:32,920 talking about machine learning. 32 00:01:32,920 --> 00:01:35,240 But in my college years, it wasn't something 33 00:01:35,240 --> 00:01:39,440 that I immersed myself that deeply into. 34 00:01:39,440 --> 00:01:43,600 It wasn't until a little later, early in my career, 35 00:01:43,600 --> 00:01:47,480 I moved to the Bay Area in 2006 to work 36 00:01:47,480 --> 00:01:50,440 at a startup called PowerSet. 37 00:01:50,440 --> 00:01:53,680 And PowerSet was a startup that was really ahead of its time. 38 00:01:53,680 --> 00:01:56,400 And that was where I first got immersed 39 00:01:56,400 --> 00:01:59,000 in the world of machine learning. 40 00:01:59,000 --> 00:02:01,720 So at PowerSet, we were oddly enough 41 00:02:01,720 --> 00:02:04,240 doing a lot of natural language processing, which 42 00:02:04,240 --> 00:02:06,280 is a hot topic these days. 43 00:02:06,280 --> 00:02:10,520 But we were using a very different approach, 44 00:02:10,520 --> 00:02:15,240 a rules-based, heuristic approach to language modeling. 45 00:02:15,240 --> 00:02:18,200 And we had licensed technology from Xerox Park 46 00:02:18,200 --> 00:02:26,240 and brought on a lot of these very learned professionals, 47 00:02:26,240 --> 00:02:30,560 PhDs in the field, tackling very hard problems 48 00:02:30,560 --> 00:02:32,320 around language understanding and how 49 00:02:32,320 --> 00:02:36,040 we could apply that to search and make a better search product. 50 00:02:36,040 --> 00:02:39,760 It was at that company I also met Lucas B. Walt, who I've now 51 00:02:39,760 --> 00:02:42,200 founded two companies with. 52 00:02:42,200 --> 00:02:48,320 And that is what really launched my career in AI and ML. 53 00:02:48,320 --> 00:02:51,760 My co-founder, Lucas, actually studied machine learning 54 00:02:51,760 --> 00:02:56,840 and had been working with models throughout college 55 00:02:56,840 --> 00:02:58,240 and in his career. 56 00:02:58,240 --> 00:03:01,160 And I'm the full-stack web developer 57 00:03:01,160 --> 00:03:04,160 that landed in this hot and exciting space 58 00:03:04,160 --> 00:03:07,800 that has had the blessing of being 59 00:03:07,800 --> 00:03:12,640 able to create tools for who I consider some of the most 60 00:03:12,640 --> 00:03:16,520 impactful and interesting engineers out there building 61 00:03:16,520 --> 00:03:19,120 the next generation of products and solutions 62 00:03:19,120 --> 00:03:20,000 on top of this stuff. 63 00:03:20,000 --> 00:03:21,440 So exciting times for sure. 64 00:03:21,440 --> 00:03:25,840 I'm glad I landed at that startup in the early 2000s. 65 00:03:25,840 --> 00:03:26,840 Awesome. 66 00:03:26,840 --> 00:03:30,040 How would you say that your background as a full-stack 67 00:03:30,040 --> 00:03:32,960 engineer sort of prepared you for the machine learning world? 68 00:03:32,960 --> 00:03:39,440 Well, I mean, I think the core thing, what I consider to be 69 00:03:39,440 --> 00:03:42,240 like most important when you're an engineer building 70 00:03:42,240 --> 00:03:45,680 products is thinking about the end user experience. 71 00:03:45,680 --> 00:03:49,120 Like how is the world going to interact with this thing? 72 00:03:49,120 --> 00:03:53,040 And I think the same is true and often a lot trickier 73 00:03:53,040 --> 00:03:55,040 with machine learning models. 74 00:03:55,040 --> 00:03:56,880 Like the second you introduce one of these models, 75 00:03:56,880 --> 00:03:59,000 you suddenly have this thing that's 76 00:03:59,000 --> 00:04:03,080 like right some percentage of the time. 77 00:04:03,080 --> 00:04:05,720 And by design, it's going to be wrong. 78 00:04:05,720 --> 00:04:11,480 So thinking about how end users are going to experience that 79 00:04:11,480 --> 00:04:16,040 or ways in which you could potentially make the end user 80 00:04:16,040 --> 00:04:19,640 experience better when they need to get involved and kind 81 00:04:19,640 --> 00:04:24,360 of handle those cases where the model is wrong, 82 00:04:24,360 --> 00:04:27,240 I think has made me hopefully more 83 00:04:27,240 --> 00:04:33,240 than maybe hopefully a better engineer and developer 84 00:04:33,240 --> 00:04:36,360 when it comes to actually bringing these machine learning 85 00:04:36,360 --> 00:04:38,520 models into the real world. 86 00:04:38,520 --> 00:04:39,520 Nice. 87 00:04:39,520 --> 00:04:41,080 Yeah, as a machine learning practitioner, 88 00:04:41,080 --> 00:04:43,720 I get to use my favorite quote like once a week, 89 00:04:43,720 --> 00:04:47,520 all models are wrong, some are useful. 90 00:04:47,520 --> 00:04:49,080 George Box, he was a statistician. 91 00:04:49,080 --> 00:04:50,400 I don't know if he was necessarily talking 92 00:04:50,400 --> 00:04:52,600 about machine learning, but it's still a fun one to get to say. 93 00:04:52,600 --> 00:04:55,480 I'm glad I got to just say it also. 94 00:04:55,480 --> 00:04:58,880 Moving forward a little bit to weights and biases, 95 00:04:58,880 --> 00:05:02,720 which is just an absolutely incredible tool. 96 00:05:02,720 --> 00:05:06,520 I've been using it for a better part of like five years 97 00:05:06,520 --> 00:05:11,400 for every part of my machine learning life cycle 98 00:05:11,400 --> 00:05:12,200 for my projects. 99 00:05:12,200 --> 00:05:13,920 I use it for a bunch of personal projects. 100 00:05:13,920 --> 00:05:16,400 I'm now using it in industry. 101 00:05:16,400 --> 00:05:20,160 Why don't you tell us in your own words as a co-founder, 102 00:05:20,160 --> 00:05:22,120 what is weights and biases? 103 00:05:22,120 --> 00:05:22,720 Yeah, you bet. 104 00:05:22,720 --> 00:05:26,080 So our mission at weights and biases 105 00:05:26,080 --> 00:05:30,080 is to build the world's best developer tools for machine 106 00:05:30,080 --> 00:05:30,960 learning engineers. 107 00:05:30,960 --> 00:05:36,320 So we're really interested in building really good tool. 108 00:05:36,320 --> 00:05:38,400 I've always been a fan of tools. 109 00:05:38,400 --> 00:05:42,160 To have a good tool, to have the right tool for the job 110 00:05:42,160 --> 00:05:47,120 in the real world is there's nothing better. 111 00:05:47,120 --> 00:05:50,600 I don't do a lot of handy work, but going to Home Depot 112 00:05:50,600 --> 00:05:52,840 and looking at the different actual tools 113 00:05:52,840 --> 00:05:55,040 is quite exhilarating for me. 114 00:05:55,040 --> 00:05:56,360 I've always enjoyed that. 115 00:05:56,360 --> 00:06:00,240 So weights and biases is building tools 116 00:06:00,240 --> 00:06:02,600 for machine learning engineers. 117 00:06:02,600 --> 00:06:07,640 So the kinds of tools that a machine learning engineer needs, 118 00:06:07,640 --> 00:06:10,560 it was pretty obvious in the early days. 119 00:06:10,560 --> 00:06:15,280 And as we've grown, it's become more nuanced. 120 00:06:15,280 --> 00:06:18,000 There's like little pockets of the problem space 121 00:06:18,000 --> 00:06:19,560 that we're always kind of going, hey, 122 00:06:19,560 --> 00:06:21,000 is there a better way to do this? 123 00:06:21,000 --> 00:06:23,200 How can create a better tool? 124 00:06:23,200 --> 00:06:24,600 Cluster problems are like, well, you 125 00:06:24,600 --> 00:06:27,280 need to keep track of what data you're training on. 126 00:06:27,280 --> 00:06:31,120 It's always when you're modeling, the data is king. 127 00:06:31,120 --> 00:06:34,640 So we created a number of tools to just have 128 00:06:34,640 --> 00:06:37,640 a solid understanding of data lineage, data versioning, 129 00:06:37,640 --> 00:06:41,720 being able to dive in and visualize, understand the data. 130 00:06:41,720 --> 00:06:43,520 And then there's a lot of experimentation 131 00:06:43,520 --> 00:06:44,280 in machine learning. 132 00:06:44,280 --> 00:06:46,280 So when you're training a machine learning model, 133 00:06:46,280 --> 00:06:48,040 it's not just the source code. 134 00:06:48,040 --> 00:06:50,320 As a traditional software developer, it's like, 135 00:06:50,320 --> 00:06:51,640 all right, I've got GitHub. 136 00:06:51,640 --> 00:06:56,480 I know what the truth is, and I have CI CD running. 137 00:06:56,480 --> 00:06:57,720 It's going to be the data, and it's 138 00:06:57,720 --> 00:07:01,000 going to be some hyperparameters, some command line arguments 139 00:07:01,000 --> 00:07:03,120 that you passed into the program that you're running. 140 00:07:03,120 --> 00:07:05,240 And then ultimately, the weights and the biases 141 00:07:05,240 --> 00:07:08,160 that you're creating when you've trained a model. 142 00:07:08,160 --> 00:07:12,160 So weights and biases is an end-to-end ML ops platform 143 00:07:12,160 --> 00:07:14,840 that helps engineers keep track of all of these things 144 00:07:14,840 --> 00:07:18,640 and then conserve as a system of record for their day-to-day 145 00:07:18,640 --> 00:07:21,680 development and understanding of how these models are 146 00:07:21,680 --> 00:07:25,720 performing and how they can make them perform better. 147 00:07:25,720 --> 00:07:27,600 Very cool. 148 00:07:27,600 --> 00:07:29,520 Yeah, the amazing thing for me is I've 149 00:07:29,520 --> 00:07:33,080 gotten to see how weights and biases has expanded over time 150 00:07:33,080 --> 00:07:35,960 and my usage of it also. 151 00:07:35,960 --> 00:07:39,680 I started out using it really just to keep track of things, 152 00:07:39,680 --> 00:07:41,840 keep track of experiment results. 153 00:07:41,840 --> 00:07:45,240 I think seeing loss curves was very illuminating for me. 154 00:07:45,240 --> 00:07:47,080 I guess I had seen it in fast AI, 155 00:07:47,080 --> 00:07:50,920 but there was something about seeing multiple runs all 156 00:07:50,920 --> 00:07:53,920 in one place, which was really nice. 157 00:07:53,920 --> 00:07:57,840 Towing around with sweeps and creating reports, all of it. 158 00:07:57,840 --> 00:07:59,560 Tables is incredible. 159 00:07:59,560 --> 00:08:03,320 Haven't gotten into Weave yet, but I'm looking forward to it. 160 00:08:03,320 --> 00:08:07,720 And ML prompts also, which is really nice. 161 00:08:07,720 --> 00:08:11,360 And speaking of, yeah, nothing better than a good tool, right? 162 00:08:11,360 --> 00:08:13,600 I mean, the right tool for the job. 163 00:08:13,600 --> 00:08:16,560 It's amazing how seamless it can be. 164 00:08:16,560 --> 00:08:19,400 And also, when you don't have the right tool, 165 00:08:19,400 --> 00:08:21,160 how frustrating it can be when you're 166 00:08:21,160 --> 00:08:27,600 trying to do something, don't try to hammer in something. 167 00:08:27,600 --> 00:08:30,320 You get a hammer, right? 168 00:08:30,320 --> 00:08:33,200 From your perspective, how have the goals of weights 169 00:08:33,200 --> 00:08:38,520 and biases changed since the onset? 170 00:08:38,520 --> 00:08:40,800 Yeah, in the beginning, right? 171 00:08:40,800 --> 00:08:44,400 This is like 2016. 172 00:08:44,400 --> 00:08:49,240 This is a time, I'm sure, much of your audience 173 00:08:49,240 --> 00:08:53,120 could remember, or maybe they were in the space at this point. 174 00:08:53,120 --> 00:08:58,000 But TensorFlow was really the main player 175 00:08:58,000 --> 00:08:59,600 in the framework space. 176 00:08:59,600 --> 00:09:02,560 PyTorch really wasn't a thing. 177 00:09:02,560 --> 00:09:06,400 Computer vision was the use case that everyone 178 00:09:06,400 --> 00:09:08,000 was talking about and excited about. 179 00:09:08,000 --> 00:09:14,320 This was a time when self-driving cars was the primary topic 180 00:09:14,320 --> 00:09:16,760 around AI or applying ML. 181 00:09:19,560 --> 00:09:22,000 The core problem that we set it to solve in the early days 182 00:09:22,000 --> 00:09:25,440 was just keeping track of your experiments. 183 00:09:25,440 --> 00:09:28,920 So state of the art at that time for just keeping 184 00:09:28,920 --> 00:09:31,800 track of your modeling effort was like a Google spreadsheet 185 00:09:31,800 --> 00:09:33,760 or an Excel spreadsheet. 186 00:09:33,760 --> 00:09:37,520 So that was a pretty low bar. 187 00:09:37,520 --> 00:09:40,480 And we just set out to make a tool that 188 00:09:40,480 --> 00:09:43,920 was really easy to keep track of the experiments 189 00:09:43,920 --> 00:09:44,960 that you're doing. 190 00:09:44,960 --> 00:09:46,640 Like originally in the very beginning, 191 00:09:46,640 --> 00:09:48,880 we didn't think putting a whole bunch of charts 192 00:09:48,880 --> 00:09:52,160 into the product was necessarily needed. 193 00:09:52,160 --> 00:09:53,400 Like the main problem we were solving 194 00:09:53,400 --> 00:09:56,400 was just keep track of the actual experiments 195 00:09:56,400 --> 00:09:58,880 and maybe what the final loss value was 196 00:09:58,880 --> 00:10:02,720 or the final accuracy value was. 197 00:10:02,720 --> 00:10:05,760 And then as we added more rich visualization features, 198 00:10:05,760 --> 00:10:10,360 we saw users love it, so we really doubled down there. 199 00:10:10,360 --> 00:10:12,880 The ways in which we've expanded was we 200 00:10:12,880 --> 00:10:15,080 finally found we convinced ourselves 201 00:10:15,080 --> 00:10:17,640 we had product market fit that we had created something that 202 00:10:17,640 --> 00:10:21,280 was useful when we got teams like OpenAI to actually use 203 00:10:21,280 --> 00:10:23,360 the product for work that they were doing 204 00:10:23,360 --> 00:10:27,440 or to a research institute on a lot of the robotics 205 00:10:27,440 --> 00:10:30,120 and autonomous vehicle work. 206 00:10:30,120 --> 00:10:32,280 So then it became like, all right, well, 207 00:10:32,280 --> 00:10:33,680 what other problems are there? 208 00:10:33,680 --> 00:10:35,080 And this was literally just going out 209 00:10:35,080 --> 00:10:37,160 talking to our customers or users 210 00:10:37,160 --> 00:10:41,720 and hearing where their pain points were. 211 00:10:41,720 --> 00:10:45,240 So the sweeps offering inside of Waste Advices 212 00:10:45,240 --> 00:10:48,720 where we make it really easy to run a hyperparameter search, 213 00:10:48,720 --> 00:10:50,160 initially it wasn't obvious. 214 00:10:50,160 --> 00:10:51,840 It was kind of like, well, there's 215 00:10:51,840 --> 00:10:54,760 good tools on the market that do that. 216 00:10:54,760 --> 00:10:57,480 We don't think we're going to magically come out 217 00:10:57,480 --> 00:11:00,120 and have the greatest hyperparameter algorithm that's 218 00:11:00,120 --> 00:11:01,240 going to save everybody money. 219 00:11:01,240 --> 00:11:04,400 It was just like, let's just make it as easy and as pleasant 220 00:11:04,400 --> 00:11:08,480 as possible to run a hyperparameter search. 221 00:11:08,480 --> 00:11:13,640 And since features like our model registry reports 222 00:11:13,640 --> 00:11:15,520 was an interesting set of features 223 00:11:15,520 --> 00:11:17,920 that came out of the reality of like, all right, well, 224 00:11:17,920 --> 00:11:22,240 everyone's report or the end result 225 00:11:22,240 --> 00:11:23,160 is very different. 226 00:11:23,160 --> 00:11:25,560 The kinds of things you want to understand and know about 227 00:11:25,560 --> 00:11:27,160 when you're doing computer vision, very different 228 00:11:27,160 --> 00:11:29,720 than if you're making a financial prediction model 229 00:11:29,720 --> 00:11:33,200 or some arbitrary classifier. 230 00:11:33,200 --> 00:11:36,640 So we built this very flexible platform 231 00:11:36,640 --> 00:11:41,920 to actually communicate these graphs and charts and results 232 00:11:41,920 --> 00:11:43,000 around to customers. 233 00:11:43,000 --> 00:11:44,880 And the product continues to evolve, 234 00:11:44,880 --> 00:11:46,800 I think most recently. 235 00:11:46,800 --> 00:11:51,360 The move from what when we started and for the past five 236 00:11:51,360 --> 00:11:53,960 years, it was always like, OK, you build a model. 237 00:11:53,960 --> 00:11:59,840 Maybe you take ResNet or some existing base model, 238 00:11:59,840 --> 00:12:01,800 but you're going to fine tune it and you're 239 00:12:01,800 --> 00:12:03,280 going to do all of this stuff in-house. 240 00:12:03,280 --> 00:12:05,480 Now it's often, we'll just call out, 241 00:12:05,480 --> 00:12:08,560 so like opening eyes API or some other API 242 00:12:08,560 --> 00:12:12,680 and the kinds of problems and things you need to be concerned 243 00:12:12,680 --> 00:12:13,360 about are different. 244 00:12:13,360 --> 00:12:15,720 But they're similar in a lot of ways as well, right? 245 00:12:15,720 --> 00:12:19,480 These are all machine systems that 246 00:12:19,480 --> 00:12:22,480 have this probabilistic nature that are going to be wrong. 247 00:12:22,480 --> 00:12:26,680 How do we evaluate and how do we try to make the user 248 00:12:26,680 --> 00:12:30,720 experience as good as possible across it? 249 00:12:30,720 --> 00:12:33,040 Yeah, absolutely. 250 00:12:33,040 --> 00:12:34,600 Yeah, there's a certain flexibility 251 00:12:34,600 --> 00:12:37,520 that's really nice with weights and biases 252 00:12:37,520 --> 00:12:43,400 that you can use it for many different use cases. 253 00:12:43,400 --> 00:12:46,440 Speaking of creating tools and sometimes you 254 00:12:46,440 --> 00:12:49,160 have the intended use for tools, what's 255 00:12:49,160 --> 00:12:51,840 a really unique use of weights and biases 256 00:12:51,840 --> 00:12:53,200 that I guess when you were creating it, 257 00:12:53,200 --> 00:12:55,960 you never really thought that it would be used for it? 258 00:12:58,400 --> 00:13:00,520 Yeah, I mean, the weights and biases platform itself, 259 00:13:00,520 --> 00:13:06,160 it's pretty versatile in terms of the core. 260 00:13:06,160 --> 00:13:08,440 As you're building a product, you're like, all right, 261 00:13:08,440 --> 00:13:12,120 well, what are the atoms of this thing? 262 00:13:12,120 --> 00:13:18,960 And I remember we built this feature a few years ago 263 00:13:18,960 --> 00:13:23,200 where we let people completely define their own visualization. 264 00:13:23,200 --> 00:13:27,080 So we built it on top of Vega, which there's 265 00:13:27,080 --> 00:13:30,920 like an altair is the Python framework that works with this 266 00:13:30,920 --> 00:13:31,880 visualization framework. 267 00:13:31,880 --> 00:13:33,640 Under the hood, it's all like D3, which 268 00:13:33,640 --> 00:13:34,920 is a very cool technology. 269 00:13:34,920 --> 00:13:38,320 But we wired up Vega such that users 270 00:13:38,320 --> 00:13:42,000 could define their own custom visualizations. 271 00:13:42,000 --> 00:13:46,160 And they could wire that up to any of the atoms 272 00:13:46,160 --> 00:13:50,640 in the weights and biases API, these units of data. 273 00:13:50,640 --> 00:13:54,440 And one of our engineers actually wired things up 274 00:13:54,440 --> 00:13:57,600 and defined a custom visualization that was actually 275 00:13:57,600 --> 00:14:02,200 like a role playing game, which I thought was awesome. 276 00:14:02,200 --> 00:14:07,440 A complete misuse of both the core Vega spec 277 00:14:07,440 --> 00:14:12,200 and the underlying data model, but a very cool demo, 278 00:14:12,200 --> 00:14:13,600 nonetheless. 279 00:14:13,600 --> 00:14:17,440 I think in the actual use cases of weights and biases, 280 00:14:17,440 --> 00:14:21,440 I've been able to see some very cool use cases of machine 281 00:14:21,440 --> 00:14:22,760 learning over the years. 282 00:14:22,760 --> 00:14:27,640 One of my favorite examples is technology around agriculture. 283 00:14:27,640 --> 00:14:30,320 So putting computer vision models 284 00:14:30,320 --> 00:14:34,360 onto big tractors and combines and reducing 285 00:14:34,360 --> 00:14:38,400 the amount of pesticides or chemicals 286 00:14:38,400 --> 00:14:44,280 that need to be applied to control weeds in a field 287 00:14:44,280 --> 00:14:46,840 has a massive impact on the environment. 288 00:14:46,840 --> 00:14:48,000 It's really cool tech. 289 00:14:48,000 --> 00:14:49,520 Like I went and saw one of the tractors, 290 00:14:49,520 --> 00:14:52,200 and they have little NVIDIA boxes 291 00:14:52,200 --> 00:14:54,680 like on the combine doing it. 292 00:14:54,680 --> 00:14:57,280 And also not one, it's not the first place your mind goes, 293 00:14:57,280 --> 00:14:59,720 where you're like, how could we use AI or ML 294 00:14:59,720 --> 00:15:02,400 to make some impact in the world? 295 00:15:02,400 --> 00:15:05,640 But yeah, the work we've done with John Deere and Blue 296 00:15:05,640 --> 00:15:08,440 River around that has been really cool to see. 297 00:15:08,440 --> 00:15:11,400 Very cool. 298 00:15:11,400 --> 00:15:13,000 In terms of all of the things that 299 00:15:13,000 --> 00:15:14,840 have been accomplished from weights and biases, 300 00:15:14,840 --> 00:15:18,520 I'm sure that you guys have a nice roadmap ahead. 301 00:15:18,520 --> 00:15:20,600 What are some of the things that you're most excited about 302 00:15:20,600 --> 00:15:24,560 for the future for weights and biases? 303 00:15:24,560 --> 00:15:26,920 Yeah, so I think the most exciting thing 304 00:15:26,920 --> 00:15:29,400 is this next generation of tooling 305 00:15:29,400 --> 00:15:34,360 for really the next generation of AI and ML engineers. 306 00:15:34,360 --> 00:15:38,800 What's happened in the last year, year and a half 307 00:15:38,800 --> 00:15:42,520 with the explosion of chat GPT, and now every data science 308 00:15:42,520 --> 00:15:44,640 conference you go to is definitely 309 00:15:44,640 --> 00:15:48,840 going to have the words like LLM or Gen AI 310 00:15:48,840 --> 00:15:50,560 somewhere on a poster. 311 00:15:50,560 --> 00:15:54,440 It's been just wild to see the whole industry 312 00:15:54,440 --> 00:16:02,120 shift to this excitement around these large models. 313 00:16:02,120 --> 00:16:03,800 The team is working on, all right, 314 00:16:03,800 --> 00:16:06,960 well, what does a product look like where you're not 315 00:16:06,960 --> 00:16:09,560 necessarily doing a lot of modeling in-house. 316 00:16:09,560 --> 00:16:12,520 You're leveraging these tools, doing more prompt engineering, 317 00:16:12,520 --> 00:16:16,960 doing more like the retrieval augmented generation space, 318 00:16:16,960 --> 00:16:19,360 kind of hooking these tools together 319 00:16:19,360 --> 00:16:27,240 and with agents and these more general purpose uses of LLM. 320 00:16:27,240 --> 00:16:30,640 It's like, what would the world's best tooling 321 00:16:30,640 --> 00:16:33,080 look like for that new world? 322 00:16:33,080 --> 00:16:40,920 That's what the team's been working on over the last year. 323 00:16:40,920 --> 00:16:46,680 And we're excited to finally release that in the next couple 324 00:16:46,680 --> 00:16:50,200 of months here and continue to iterate on it. 325 00:16:50,200 --> 00:16:53,440 As we found with our existing product, 326 00:16:53,440 --> 00:16:57,800 it's like we make a swing, we try to make something as good 327 00:16:57,800 --> 00:17:01,360 as we think it can be, and then through actually having people 328 00:17:01,360 --> 00:17:05,720 use it and solve problems, we can iterate and make 329 00:17:05,720 --> 00:17:06,840 it great and delightful. 330 00:17:06,840 --> 00:17:10,720 So that's really the area we're focusing a lot on. 331 00:17:10,720 --> 00:17:12,560 I think one of the big shifts there 332 00:17:12,560 --> 00:17:15,240 is that from the start of the company, 333 00:17:15,240 --> 00:17:18,600 we're selling a product to machine learning engineers. 334 00:17:18,600 --> 00:17:21,080 These are people that understand the underlying math. 335 00:17:21,080 --> 00:17:24,920 They understand probabilities and what 336 00:17:24,920 --> 00:17:29,520 that means from an operational standpoint. 337 00:17:29,520 --> 00:17:31,880 In this new world, we have a lot of just traditional software 338 00:17:31,880 --> 00:17:34,840 developers that are now consuming these APIs 339 00:17:34,840 --> 00:17:36,920 and building products on top of them. 340 00:17:36,920 --> 00:17:39,160 So one of the challenges is, how do we 341 00:17:39,160 --> 00:17:47,160 convey these core ideas to this new audience in a way that 342 00:17:47,160 --> 00:17:53,960 enables them to build better products without a lot of the. 343 00:17:53,960 --> 00:17:55,640 There's a lot of new, it's tricky. 344 00:17:55,640 --> 00:17:57,160 There's going to potentially be bias. 345 00:17:57,160 --> 00:17:59,080 You need to really think carefully about, OK, 346 00:17:59,080 --> 00:18:01,520 when this thing fails, how's it going to fail? 347 00:18:01,520 --> 00:18:06,240 You want to fail in a way that's least disruptive 348 00:18:06,240 --> 00:18:08,800 to the end user. 349 00:18:08,800 --> 00:18:11,840 So being able to build tools for this space, 350 00:18:11,840 --> 00:18:13,040 it's really exciting. 351 00:18:13,040 --> 00:18:15,920 And there's hundreds of other companies doing the same thing. 352 00:18:15,920 --> 00:18:18,280 So we've got a lot of work to do, and we need to do quickly. 353 00:18:18,280 --> 00:18:20,720 But it's an exciting space to be in. 354 00:18:20,720 --> 00:18:23,880 Yeah, that's definitely one of the most challenging things 355 00:18:23,880 --> 00:18:26,600 with machine learning versus, like, say, 356 00:18:26,600 --> 00:18:27,920 traditional programming. 357 00:18:27,920 --> 00:18:29,760 If it doesn't work for traditional programming, 358 00:18:29,760 --> 00:18:31,080 you just get an error, right? 359 00:18:31,080 --> 00:18:32,320 I mean, usually, most of the time, 360 00:18:32,320 --> 00:18:34,120 unless it's something really weird. 361 00:18:34,120 --> 00:18:37,640 But machine learning, you'll get an answer. 362 00:18:37,640 --> 00:18:40,160 But it won't be right. 363 00:18:40,160 --> 00:18:45,640 And with an API call, you will generate text, 364 00:18:45,640 --> 00:18:47,600 you will generate some image, but will it 365 00:18:47,600 --> 00:18:52,120 be useful for what you're actually trying to do? 366 00:18:52,120 --> 00:18:57,080 And understanding the, I guess, the responsibility 367 00:18:57,080 --> 00:18:59,480 that people have when they're creating things like that, 368 00:18:59,480 --> 00:19:02,200 it's a real transition. 369 00:19:02,200 --> 00:19:03,400 And it's tempting. 370 00:19:03,400 --> 00:19:06,280 You can make a cool demo today. 371 00:19:06,280 --> 00:19:09,160 Like, it's been so fun as an engineer having access 372 00:19:09,160 --> 00:19:12,880 to this technology and to delight myself when I make something 373 00:19:12,880 --> 00:19:13,960 and I'm like, whoa, it did that. 374 00:19:13,960 --> 00:19:16,680 I can't believe it did it. 375 00:19:16,680 --> 00:19:19,440 But that demo, where you then kind of script it 376 00:19:19,440 --> 00:19:23,640 and you're showing your friends this cool thing you made, 377 00:19:23,640 --> 00:19:28,280 it does not account for all of the weird edge cases 378 00:19:28,280 --> 00:19:30,320 and things you haven't thought about in ways in which 379 00:19:30,320 --> 00:19:33,600 another user is going to interact with this thing. 380 00:19:33,600 --> 00:19:35,280 And if you just throw that out there, 381 00:19:35,280 --> 00:19:38,080 you're not even going to know really if it's working or not. 382 00:19:38,080 --> 00:19:40,280 The closest proxy you'll have is like, 383 00:19:40,280 --> 00:19:42,640 are people sharing it and more people using it. 384 00:19:42,640 --> 00:19:45,400 But even thinking about, all right, well, 385 00:19:45,400 --> 00:19:49,040 how do I get user feedback? 386 00:19:49,040 --> 00:19:53,400 I mean, this goes back to that first job I had here 387 00:19:53,400 --> 00:19:56,680 in the valley, getting into machine learning. 388 00:19:56,680 --> 00:19:58,440 When it's a search engine, how do you 389 00:19:58,440 --> 00:20:01,800 know if you're a machine learning algorithm that return 390 00:20:01,800 --> 00:20:03,520 results is any good? 391 00:20:03,520 --> 00:20:08,800 Well, a good proxy is like, are people clicking on the results? 392 00:20:08,800 --> 00:20:12,480 But it's a subtle gnarly problem. 393 00:20:12,480 --> 00:20:14,440 And you need to really think about it 394 00:20:14,440 --> 00:20:17,320 and have rigorous ways to evaluate and understand 395 00:20:17,320 --> 00:20:19,000 if you're getting better or worse, because you're 396 00:20:19,000 --> 00:20:20,200 going to have to change the prompt. 397 00:20:20,200 --> 00:20:21,400 You're going to upgrade the model. 398 00:20:21,400 --> 00:20:23,360 You're going to change things about your product. 399 00:20:23,360 --> 00:20:26,240 And you need a way to actually measure, 400 00:20:26,240 --> 00:20:30,320 is this thing good or bad without just sending it out 401 00:20:30,320 --> 00:20:33,200 to your users and making them kind of yell, hey, what the heck? 402 00:20:33,200 --> 00:20:34,880 This sucks. 403 00:20:34,880 --> 00:20:36,200 Yeah, for sure. 404 00:20:36,200 --> 00:20:40,280 And speaking of the ability to create demos, 405 00:20:40,280 --> 00:20:42,120 maybe I'm not sure if it's over said or anything, 406 00:20:42,120 --> 00:20:43,880 but something I've been finding myself saying, 407 00:20:43,880 --> 00:20:45,880 it's easy to create a demo. 408 00:20:45,880 --> 00:20:48,640 It's hard to create something for production. 409 00:20:48,640 --> 00:20:50,920 And it's even harder to create something at scale. 410 00:20:50,920 --> 00:20:52,600 Something can work a dozen times. 411 00:20:52,600 --> 00:20:54,440 But is it going to work a thousand times? 412 00:20:54,440 --> 00:20:56,040 How's it going to work a million times? 413 00:20:56,040 --> 00:20:57,960 How's it going to work when there's multiple users 414 00:20:57,960 --> 00:20:58,840 at the same time? 415 00:20:58,840 --> 00:21:01,600 How's it going to work on all of these edge cases? 416 00:21:01,600 --> 00:21:04,600 And I think that what we're seeing 417 00:21:04,600 --> 00:21:06,920 is that especially with this generative AI, 418 00:21:06,920 --> 00:21:09,280 you can't even test all of these things. 419 00:21:09,280 --> 00:21:12,720 You can't even fully check it for prompt injection, 420 00:21:12,720 --> 00:21:15,520 let's say, because until it's out there 421 00:21:15,520 --> 00:21:19,000 and people are starting to use it for these unintended uses, 422 00:21:19,000 --> 00:21:21,880 that's when you start to see all these crazy things come out. 423 00:21:21,880 --> 00:21:23,640 But it's already kind of too late, 424 00:21:23,640 --> 00:21:25,360 because it's in production. 425 00:21:25,360 --> 00:21:26,160 Someone is using it. 426 00:21:26,160 --> 00:21:30,240 They have, it's already exposed. 427 00:21:30,240 --> 00:21:31,400 It's already out. 428 00:21:31,400 --> 00:21:33,120 We're seeing lots of things like that happen 429 00:21:33,120 --> 00:21:35,440 where people are putting out generative chatbots 430 00:21:35,440 --> 00:21:37,120 for their customer service. 431 00:21:37,120 --> 00:21:39,880 And it's just like, that's a terrible idea 432 00:21:40,880 --> 00:21:43,920 to do that fully, to just be fully relying on that. 433 00:21:43,920 --> 00:21:46,040 And there's obviously other examples too. 434 00:21:47,640 --> 00:21:49,880 But yeah, speaking of evaluation, 435 00:21:49,880 --> 00:21:51,280 it's really hard. 436 00:21:51,280 --> 00:21:55,640 How do you know if your product is working correctly? 437 00:21:55,640 --> 00:21:59,720 So yeah, something like search, it's very difficult, right? 438 00:21:59,720 --> 00:22:02,000 You might want, you could quickly get results, 439 00:22:02,000 --> 00:22:03,600 but are they the right results? 440 00:22:04,880 --> 00:22:06,440 Recommendation engines, right? 441 00:22:06,440 --> 00:22:07,800 You can quickly get results, 442 00:22:07,800 --> 00:22:09,680 but are they the right results? 443 00:22:09,680 --> 00:22:12,560 I think evaluation will always remain a problem, 444 00:22:14,320 --> 00:22:18,040 especially because I think people put too much weight 445 00:22:18,040 --> 00:22:19,680 on benchmarks as well. 446 00:22:21,120 --> 00:22:23,400 I don't know what you're feeling is on that. 447 00:22:23,400 --> 00:22:24,240 You think about that one? 448 00:22:24,240 --> 00:22:25,320 The benchmarks are very generic, right? 449 00:22:25,320 --> 00:22:27,480 So then, some will make an announcement and say, 450 00:22:27,480 --> 00:22:32,480 hey, we're better than GPT-4 in like MMLU or, 451 00:22:34,680 --> 00:22:37,080 I'm not even sure if that's one of the correct acronyms 452 00:22:37,080 --> 00:22:40,720 of the 30 core tests that people are throwing out there. 453 00:22:42,760 --> 00:22:45,680 And there's not, those are important. 454 00:22:45,680 --> 00:22:50,680 It's good to have some general set of benchmarks 455 00:22:50,720 --> 00:22:54,400 for different things that we're testing, 456 00:22:54,400 --> 00:22:56,840 but they're very general and they're never gonna tell you 457 00:22:56,840 --> 00:22:59,800 how good is this thing gonna be for my specific use case. 458 00:22:59,800 --> 00:23:03,560 You're the only one who can answer that question. 459 00:23:03,560 --> 00:23:05,600 And it could be hard to answer it. 460 00:23:05,600 --> 00:23:09,920 So like going back to the search engine ranking algorithm, 461 00:23:11,240 --> 00:23:12,080 well, how do you do that? 462 00:23:12,080 --> 00:23:16,400 Well, it turns out you hire a bunch of people 463 00:23:16,400 --> 00:23:19,400 who are trained often with a big manual 464 00:23:19,400 --> 00:23:23,160 on here's how we define relevance, 465 00:23:23,160 --> 00:23:25,120 which is already a pretty fuzzy subject, 466 00:23:25,120 --> 00:23:28,880 like how relevant is something to a given user's query. 467 00:23:28,880 --> 00:23:30,400 And then you have them label the data. 468 00:23:30,400 --> 00:23:32,720 You look at a whole bunch of queries and results 469 00:23:32,720 --> 00:23:35,000 and you have them on a scale of like one to four 470 00:23:35,000 --> 00:23:39,080 or one to five, say how relevant a given result is 471 00:23:39,080 --> 00:23:41,600 for a query and even then you're like, okay, well, 472 00:23:41,600 --> 00:23:44,880 you have to, the query could be ambiguous. 473 00:23:44,880 --> 00:23:47,360 It's hard to understand what a user's intent is 474 00:23:47,360 --> 00:23:48,720 when they query. 475 00:23:48,720 --> 00:23:53,000 These problems are very similar in the chat space 476 00:23:53,000 --> 00:23:55,960 or having a user ask for something. 477 00:23:55,960 --> 00:23:58,440 And then when the ultimate result comes out, 478 00:23:58,440 --> 00:24:01,040 you have to, you need some way to measure, 479 00:24:01,040 --> 00:24:05,600 well, okay, did this satisfy the user's question? 480 00:24:05,600 --> 00:24:07,800 We see it in chat GPT itself. 481 00:24:07,800 --> 00:24:11,400 We can give a little thumbs up or thumbs down. 482 00:24:11,400 --> 00:24:15,360 Most people probably don't interact in that way. 483 00:24:15,360 --> 00:24:18,040 When a user does, it's a really strong signal. 484 00:24:18,040 --> 00:24:20,120 Right, so you should probably incorporate that data 485 00:24:20,120 --> 00:24:25,120 back into your process and use it to make the model better. 486 00:24:26,360 --> 00:24:29,080 But yeah, I mean, the good news is companies 487 00:24:29,080 --> 00:24:33,240 have been working on this problem for 20 years. 488 00:24:34,560 --> 00:24:38,840 The bad news is every individual has like a slightly 489 00:24:38,840 --> 00:24:42,440 different definition of good for whatever they're doing. 490 00:24:42,440 --> 00:24:45,480 So there isn't just this magic, I can buy this product 491 00:24:45,480 --> 00:24:47,880 and it's gonna like solve this problem for me. 492 00:24:47,880 --> 00:24:51,200 What you need, you need like really good tools to help you 493 00:24:51,200 --> 00:24:53,120 ask the question and solve the problem, 494 00:24:53,120 --> 00:24:55,400 which is why we built weights and biases 495 00:24:55,400 --> 00:24:58,280 and hope we can really help a lot of people 496 00:24:59,440 --> 00:25:02,320 put this rigorous process in place to be able to build 497 00:25:02,320 --> 00:25:05,640 a robust data science machine learning function. 498 00:25:05,640 --> 00:25:06,920 Yeah, absolutely. 499 00:25:07,880 --> 00:25:10,800 One of the things that weights and biases has helped me, 500 00:25:10,800 --> 00:25:14,120 it's like you try to get this one metric, right? 501 00:25:14,120 --> 00:25:15,720 Like you try to get like, oh, okay, 502 00:25:15,720 --> 00:25:17,640 F1 score is above 0.8, right? 503 00:25:17,640 --> 00:25:20,120 But it doesn't really matter that much. 504 00:25:20,120 --> 00:25:21,880 It's sometimes it's about how it's performing 505 00:25:21,880 --> 00:25:24,520 on different segments of your data. 506 00:25:24,520 --> 00:25:26,960 And I found that tables has helped me a lot. 507 00:25:26,960 --> 00:25:30,240 I've been able to look at different probability distributions 508 00:25:30,240 --> 00:25:34,000 for different classes and also just to see where there's, 509 00:25:34,000 --> 00:25:37,120 to see where there's errors and to sort of segment the data 510 00:25:37,120 --> 00:25:40,480 and then see, okay, in this particular type of conversation 511 00:25:40,480 --> 00:25:44,880 that I'm analyzing, you know, this is what I wanna be looking 512 00:25:44,880 --> 00:25:46,800 for, okay, I need to, these are like, 513 00:25:46,800 --> 00:25:48,960 it helps me with error analysis basically 514 00:25:48,960 --> 00:25:51,880 and to zoom in on those problems. 515 00:25:51,880 --> 00:25:55,760 Because often, yeah, it's not just about one accuracy metric 516 00:25:55,760 --> 00:25:59,880 or one particular thing, you have to sort of have this ability 517 00:25:59,880 --> 00:26:01,520 to zoom in and zoom out. 518 00:26:01,520 --> 00:26:03,440 And that's one thing like weights and biases 519 00:26:03,440 --> 00:26:05,400 has really helped me with. 520 00:26:05,400 --> 00:26:06,760 Yeah, I mean, this idea is like, you know, 521 00:26:06,760 --> 00:26:11,360 a confusion matrix of like, I'm making a model to predict 522 00:26:11,360 --> 00:26:13,400 like whether or not you have COVID. 523 00:26:13,400 --> 00:26:17,680 Like if it, like false positives versus false negative, 524 00:26:17,680 --> 00:26:19,760 it's like different for the use case. 525 00:26:19,760 --> 00:26:22,320 Like I would, if I tell someone they have COVID 526 00:26:22,320 --> 00:26:25,160 and they don't actually have it, probably not as bad 527 00:26:25,160 --> 00:26:27,120 as me telling someone they don't have COVID 528 00:26:27,120 --> 00:26:28,520 when they actually have it. 529 00:26:28,520 --> 00:26:29,360 Right. 530 00:26:29,360 --> 00:26:31,880 So how do you wanna optimize your model for these cases? 531 00:26:31,880 --> 00:26:35,440 What can you do to really prevent that? 532 00:26:35,440 --> 00:26:38,720 Like the case you don't want, right? 533 00:26:38,720 --> 00:26:42,720 These broad like F1 score 80%, yeah, it means nothing. 534 00:26:42,720 --> 00:26:45,960 How many times am I gonna be like lying to my user 535 00:26:45,960 --> 00:26:48,240 about this thing that's really important? 536 00:26:48,240 --> 00:26:49,160 Right. 537 00:26:49,160 --> 00:26:50,000 Yeah. 538 00:26:50,000 --> 00:26:52,960 It's whenever the cost of errors aren't equal 539 00:26:52,960 --> 00:26:55,720 and it's always that case, right? 540 00:26:55,720 --> 00:26:58,760 Cause the cost of errors are never the same. 541 00:26:59,840 --> 00:27:04,200 So therefore the metric can't just be this overall metric 542 00:27:04,200 --> 00:27:06,720 where you're treating true positives and false positives 543 00:27:06,720 --> 00:27:09,520 or whatever, you know, true negatives and whatever. 544 00:27:09,520 --> 00:27:13,120 All of your combinations in your confusion matrix, 545 00:27:13,120 --> 00:27:15,480 each box matters differently 546 00:27:15,480 --> 00:27:17,920 and you have to be able to somehow incorporate that. 547 00:27:17,920 --> 00:27:19,440 And the only way you can really do that 548 00:27:19,440 --> 00:27:21,960 is by, you know, segmenting it. 549 00:27:21,960 --> 00:27:24,280 Especially when you're iterating, like, you know, 550 00:27:24,280 --> 00:27:28,080 maybe I moved F1 score from 80% to 90%. 551 00:27:29,280 --> 00:27:31,240 That's a no, of course, let's ship that model. 552 00:27:31,240 --> 00:27:33,960 Well, wait, like look at those cases. 553 00:27:33,960 --> 00:27:35,760 Did the cases get better or worse? 554 00:27:35,760 --> 00:27:37,280 Cause maybe overall you got better, 555 00:27:37,280 --> 00:27:39,800 but now you're like way worse on the false positive 556 00:27:39,800 --> 00:27:40,800 or whatever case. 557 00:27:40,800 --> 00:27:42,600 And that's really important to know. 558 00:27:42,600 --> 00:27:43,440 Right. 559 00:27:43,440 --> 00:27:45,720 Or you'll just get better at the majority class 560 00:27:45,720 --> 00:27:48,680 and then you won't even ever detect the rare class 561 00:27:48,680 --> 00:27:52,000 and you'll think, oh, okay, yeah, my model's better. 562 00:27:52,000 --> 00:27:54,080 I know people just wanna know is this, 563 00:27:54,080 --> 00:27:55,600 is model A better than model B, 564 00:27:55,600 --> 00:27:57,640 but there's always some trade off. 565 00:27:57,640 --> 00:28:00,160 It's never, very rarely do you ever get it 566 00:28:00,160 --> 00:28:02,880 like across the board that one thing is better, 567 00:28:02,880 --> 00:28:07,240 you know, categorically better than another model. 568 00:28:07,240 --> 00:28:10,160 You know, I mean, like these models are gonna get better. 569 00:28:10,160 --> 00:28:11,520 They're gonna do more amazing things. 570 00:28:11,520 --> 00:28:14,640 It's an exciting time for us to be in. 571 00:28:14,640 --> 00:28:17,840 But as these models get generally better, 572 00:28:17,840 --> 00:28:21,160 this problem of like, all right, well, when it fails, 573 00:28:21,160 --> 00:28:24,040 knowing how it fails and doing everything we can 574 00:28:24,040 --> 00:28:25,880 to like inform the user and protect against it, 575 00:28:25,880 --> 00:28:28,440 it's gonna become even bigger. 576 00:28:28,440 --> 00:28:31,040 Cause we're gonna start trusting these things more. 577 00:28:31,040 --> 00:28:33,360 Like I bet we'll never get rid of hallucination 578 00:28:33,360 --> 00:28:37,560 because by definition of the way these things work, 579 00:28:37,560 --> 00:28:41,080 there's some weird corner case or something weird 580 00:28:41,080 --> 00:28:44,120 with the data that's gonna like be really bad. 581 00:28:44,120 --> 00:28:45,600 It's very important to understand that 582 00:28:45,600 --> 00:28:48,040 and do it we can to prevent users 583 00:28:48,040 --> 00:28:52,200 from having a bad experience because of it. 584 00:28:52,200 --> 00:28:54,760 Yeah, 100%. 585 00:28:54,760 --> 00:28:57,360 Yeah, I know, I always find it so funny 586 00:28:57,360 --> 00:29:02,000 like companies say we have eliminated hallucinations. 587 00:29:02,000 --> 00:29:05,400 If you've said that, then don't trust that company 588 00:29:05,400 --> 00:29:06,720 because they don't know what they're talking about. 589 00:29:06,720 --> 00:29:08,240 It's like eliminating bias. 590 00:29:08,240 --> 00:29:10,400 It's like, no, you have not eliminated bias. 591 00:29:10,400 --> 00:29:13,400 You can try to minimize it, but you cannot eliminate it. 592 00:29:13,400 --> 00:29:15,280 And if you think that you have, 593 00:29:15,280 --> 00:29:18,040 then you didn't really fully think through your problem. 594 00:29:18,960 --> 00:29:19,800 Yeah. 595 00:29:20,880 --> 00:29:24,160 So just like looking at this space and, you know, 596 00:29:24,160 --> 00:29:26,160 obviously like the last year and a half 597 00:29:26,160 --> 00:29:29,000 has been this hype cycle, right? 598 00:29:29,000 --> 00:29:31,040 But you've been in this industry, you know, 599 00:29:31,040 --> 00:29:35,000 since like 2007, were there any other like big 600 00:29:36,400 --> 00:29:39,720 revolutionary like step function things like this 601 00:29:39,720 --> 00:29:41,960 that really created such hype? 602 00:29:41,960 --> 00:29:43,720 Have you ever seen something like this 603 00:29:43,720 --> 00:29:45,400 like chatGPT has created? 604 00:29:47,240 --> 00:29:48,200 Not to this level. 605 00:29:48,200 --> 00:29:52,760 I mean, this is astronomical hype and it like continues. 606 00:29:52,760 --> 00:29:55,400 I kind of thought like, all right, people will chill. 607 00:29:56,800 --> 00:29:59,120 But there's still like every conference I go to, 608 00:29:59,120 --> 00:30:03,760 every company I talk to, they're, you know, 609 00:30:03,760 --> 00:30:05,680 deploying a lot of resources to figure out 610 00:30:05,680 --> 00:30:10,000 how generative AI is gonna change how they function, 611 00:30:10,000 --> 00:30:10,960 how the world functions. 612 00:30:10,960 --> 00:30:15,960 So this is definitely unlike anything I've ever experienced. 613 00:30:16,200 --> 00:30:21,200 The closest is maybe the, yeah, 614 00:30:22,480 --> 00:30:24,480 the hype around autonomous vehicles. 615 00:30:24,480 --> 00:30:27,520 Really like when we first started weights and biases, 616 00:30:27,520 --> 00:30:29,440 it was clear that, okay, deep learning was really 617 00:30:29,440 --> 00:30:31,440 starting to work. 618 00:30:31,440 --> 00:30:35,320 Like the things that the demos I was seeing, 619 00:30:35,320 --> 00:30:40,240 how good these models were getting at just taking in pixels 620 00:30:40,240 --> 00:30:43,840 and spitting out like what everything in that image was 621 00:30:43,840 --> 00:30:47,360 or putting bounding boxes around important objects was, 622 00:30:47,360 --> 00:30:49,360 I remember seeing examples of it being like, wow, 623 00:30:49,360 --> 00:30:52,160 I did not think we'd be able to do this 624 00:30:52,160 --> 00:30:54,000 when we were able to do it. 625 00:30:54,000 --> 00:30:54,840 Right. 626 00:30:54,840 --> 00:30:59,440 And I think you saw, you know, a ton of money go into 627 00:30:59,440 --> 00:31:01,480 a ton of different companies trying to make 628 00:31:01,480 --> 00:31:05,720 self-driven cars and predictions of having a self-driven car 629 00:31:05,720 --> 00:31:10,200 before, you know, well before we actually were able to have it. 630 00:31:10,200 --> 00:31:13,400 But, you know, now I'm going around streets of San Francisco 631 00:31:13,400 --> 00:31:17,400 and seeing the ways cars drive by without someone in them 632 00:31:17,400 --> 00:31:19,240 or taking rides in them, which is trippy. 633 00:31:19,240 --> 00:31:20,600 Like you've been in it. 634 00:31:20,600 --> 00:31:21,440 It's here. 635 00:31:21,440 --> 00:31:26,120 It's a little bit longer than any of us had hoped, 636 00:31:26,120 --> 00:31:28,240 but it's here. 637 00:31:28,240 --> 00:31:29,080 You've taken one? 638 00:31:29,080 --> 00:31:32,000 I think, yeah, yeah, a couple of times. 639 00:31:32,000 --> 00:31:33,000 It's cool. 640 00:31:33,000 --> 00:31:33,840 Creepy? 641 00:31:33,840 --> 00:31:35,000 It's very cool. 642 00:31:35,000 --> 00:31:36,600 Yeah, definitely a little creepy. 643 00:31:39,320 --> 00:31:42,320 And I've seen it's gotten into, I love like writing in ways 644 00:31:42,320 --> 00:31:44,520 because you like see some situation and you'll be like, 645 00:31:44,520 --> 00:31:46,320 I want to get like a bag of popcorn and be like, 646 00:31:46,320 --> 00:31:47,800 what's it going to do here? 647 00:31:47,800 --> 00:31:50,440 We've got like construction codes. 648 00:31:50,440 --> 00:31:52,560 Homeless person doing something crazy. 649 00:31:52,560 --> 00:31:53,760 Let's like see. 650 00:31:53,760 --> 00:31:56,680 I've always been pleasantly surprised. 651 00:31:56,680 --> 00:31:58,480 Right, yeah. 652 00:31:58,480 --> 00:32:01,320 Yeah, I don't know. 653 00:32:01,320 --> 00:32:03,280 It's creepy. 654 00:32:03,280 --> 00:32:05,800 Are there steering wheels or there's no steering wheel? 655 00:32:05,800 --> 00:32:07,040 Yeah, there's a steering wheel. 656 00:32:07,040 --> 00:32:09,160 You can even sit in the driver's seat. 657 00:32:09,160 --> 00:32:12,200 Apparently you have to keep your hands off of the, 658 00:32:12,200 --> 00:32:13,880 I haven't done that. 659 00:32:13,880 --> 00:32:14,880 Yeah. 660 00:32:14,880 --> 00:32:16,560 I get in usually in the back seat or something 661 00:32:16,560 --> 00:32:18,960 and I'll take like a video because I'm still, you know, 662 00:32:18,960 --> 00:32:21,920 when you see the wheel turning and it's going. 663 00:32:21,920 --> 00:32:24,080 Yeah, it's pretty cool. 664 00:32:24,080 --> 00:32:25,440 I guess it works. 665 00:32:25,440 --> 00:32:28,360 It needs to stay within a certain area though, right? 666 00:32:28,360 --> 00:32:30,280 It can't go outside of a certain area. 667 00:32:30,280 --> 00:32:31,320 Is that how it is? 668 00:32:32,880 --> 00:32:34,960 It takes some weird routes. 669 00:32:34,960 --> 00:32:35,800 Oh, okay. 670 00:32:35,800 --> 00:32:36,920 Like it's definitely like its route planner 671 00:32:36,920 --> 00:32:39,640 is not just like Google Maps. 672 00:32:39,640 --> 00:32:40,840 Yeah. 673 00:32:40,840 --> 00:32:43,480 But yeah, I don't know how they license it with the city 674 00:32:43,480 --> 00:32:46,280 or if there's certain like no go zones. 675 00:32:46,280 --> 00:32:50,200 But they also like the tech on those things is nuts. 676 00:32:50,200 --> 00:32:53,480 That is not a cheap vehicle to operate 677 00:32:53,480 --> 00:32:56,280 and there's lots of light ours and all these things 678 00:32:56,280 --> 00:32:58,320 that Elon doesn't like. 679 00:32:58,320 --> 00:32:59,960 But you know, it turns out it makes the problem 680 00:32:59,960 --> 00:33:02,480 a lot more doable. 681 00:33:02,480 --> 00:33:03,720 But yeah. 682 00:33:03,720 --> 00:33:05,880 Take in whatever senses you need to take in 683 00:33:05,880 --> 00:33:06,840 to get that done. 684 00:33:06,840 --> 00:33:10,680 You don't have to have it be some all knowing 685 00:33:10,680 --> 00:33:12,640 omniscient sort of model. 686 00:33:12,640 --> 00:33:17,520 It can take in multiple senses. Yeah, that's cool. 687 00:33:17,520 --> 00:33:19,680 I need to look into it even more. 688 00:33:19,680 --> 00:33:21,680 I don't know if I would take it or not. 689 00:33:21,680 --> 00:33:24,000 I guess eventually that'll become commonplace. 690 00:33:24,000 --> 00:33:25,960 You do it enough, you'll be exposed to it. 691 00:33:25,960 --> 00:33:29,640 You'll be, you'll stop taking, you know, stop taking videos. 692 00:33:29,640 --> 00:33:31,000 Come on, you know, it's exciting man. 693 00:33:31,000 --> 00:33:33,320 It's, you should take it. 694 00:33:33,320 --> 00:33:34,160 Yeah. 695 00:33:34,160 --> 00:33:35,720 I'll come to San Francisco. 696 00:33:35,720 --> 00:33:37,240 I'll get you a ride in one. 697 00:33:37,240 --> 00:33:38,080 I appreciate it. 698 00:33:38,080 --> 00:33:42,880 I would take a ride with you in a driverless car. 699 00:33:42,880 --> 00:33:45,480 I would do it. 700 00:33:45,480 --> 00:33:46,600 Very cool. 701 00:33:46,600 --> 00:33:50,560 So with all of this hype and everything 702 00:33:50,560 --> 00:33:53,480 that's happening in, you know, let's say natural language 703 00:33:53,480 --> 00:33:56,560 processing, but really just like the machine learning world, 704 00:33:56,560 --> 00:34:01,280 how do you view the gap between the hype and the reality? 705 00:34:01,280 --> 00:34:03,920 So like what the promise is of all of this stuff 706 00:34:03,920 --> 00:34:08,040 and then like where we actually are? 707 00:34:08,040 --> 00:34:08,880 Yeah. 708 00:34:08,880 --> 00:34:13,480 Well, like I said, I'm surprised that the, like, 709 00:34:13,480 --> 00:34:16,720 where we're still like peak hype from what I can see. 710 00:34:16,720 --> 00:34:18,520 So, you know, we're going to reach, 711 00:34:18,520 --> 00:34:21,080 we're going to hit the trough of disillusionment 712 00:34:21,080 --> 00:34:21,720 at some point. 713 00:34:21,720 --> 00:34:23,400 This is the, you know, the Gartner hype cycle. 714 00:34:26,240 --> 00:34:35,680 I think, you know, a big gap, like this space moves so fast. 715 00:34:35,680 --> 00:34:37,640 You know, waste and biases has been around five years. 716 00:34:37,640 --> 00:34:41,040 The amount of change, you know, the transformer architecture, 717 00:34:41,040 --> 00:34:43,520 for instance, like wasn't a thing until 2017. 718 00:34:43,520 --> 00:34:47,840 And now that's basically the most popular architecture used 719 00:34:47,840 --> 00:34:51,240 in everything from the self-driven cars 720 00:34:51,240 --> 00:34:55,280 to these language models. 721 00:34:55,280 --> 00:34:58,760 And, you know, I'm sure there'll be another architecture 722 00:34:58,760 --> 00:35:00,640 or changes to this architecture that 723 00:35:00,640 --> 00:35:03,160 proved to be even more fruitful. 724 00:35:03,160 --> 00:35:12,160 So, the, yeah, well, I think the speed is jarring. 725 00:35:12,160 --> 00:35:16,480 And then when you get these big enterprise companies 726 00:35:16,480 --> 00:35:19,840 figuring out how to use this new thing, they're slow. 727 00:35:19,840 --> 00:35:23,160 Like they're still, you know, very much being cautious 728 00:35:23,160 --> 00:35:26,400 and figuring it out. 729 00:35:26,400 --> 00:35:28,120 And, you know, we're just sitting, 730 00:35:28,120 --> 00:35:30,840 we're waiting for the number of transistors 731 00:35:30,840 --> 00:35:35,600 that NVIDIA can pack into their gyps to go up, which it will. 732 00:35:35,600 --> 00:35:38,000 And then these models will get better. 733 00:35:38,000 --> 00:35:43,840 And I saw, there was like an interview with Sam Altman, 734 00:35:43,840 --> 00:35:45,960 saying a lot of people think, like, oh, we'll get this, 735 00:35:45,960 --> 00:35:50,040 like, AGI or even the couple weeks after chat GPT blew up, 736 00:35:50,040 --> 00:35:51,760 everyone was like, oh, my god, this is going to, like, 737 00:35:51,760 --> 00:35:53,520 change everything now. 738 00:35:53,520 --> 00:35:55,360 It takes time. 739 00:35:55,360 --> 00:35:59,360 It is the actual process of finding the killer use cases 740 00:35:59,360 --> 00:36:05,720 for this and making it a core part of what you're doing. 741 00:36:05,720 --> 00:36:07,600 It will take time. 742 00:36:07,600 --> 00:36:10,560 I think, well, you look at, like, why 743 00:36:10,560 --> 00:36:13,400 Combinator and the startups coming out of that now, 744 00:36:13,400 --> 00:36:19,600 like, the majority are somehow connected to this space. 745 00:36:24,160 --> 00:36:26,720 What was the original question? 746 00:36:26,720 --> 00:36:28,080 What are the challenges going to be? 747 00:36:28,080 --> 00:36:29,840 Yeah. 748 00:36:29,840 --> 00:36:34,600 No, the gap between the hype and the reality. 749 00:36:34,600 --> 00:36:38,160 Yeah, I mean, I think this is self-serving. 750 00:36:38,160 --> 00:36:40,120 One of the big gaps is just better tooling, 751 00:36:40,120 --> 00:36:44,640 like, having visibility into how these things are performing 752 00:36:44,640 --> 00:36:46,400 and actually operationalizing it. 753 00:36:49,040 --> 00:36:51,360 You know, I think that's the thing that's happened is, 754 00:36:51,360 --> 00:36:52,600 you can use like GPT-4. 755 00:36:52,600 --> 00:36:53,960 It does these amazing things. 756 00:36:53,960 --> 00:36:57,600 But it's slow, and it's expensive at scale. 757 00:36:57,600 --> 00:36:59,200 So then people are, all right, well, yeah, 758 00:36:59,200 --> 00:37:02,480 we'll take Lama 2 and find, well, now you 759 00:37:02,480 --> 00:37:05,800 need to have a robust like MLOps process and practice 760 00:37:05,800 --> 00:37:08,560 to iterate on that model and understand its shortcomings 761 00:37:08,560 --> 00:37:13,600 and prevent all of these safety-related issues. 762 00:37:13,600 --> 00:37:16,040 So I think the gap now is that, yeah, there 763 00:37:16,040 --> 00:37:18,760 aren't a lot of push-button-managed solutions 764 00:37:18,760 --> 00:37:20,080 out there. 765 00:37:20,080 --> 00:37:21,680 Often, the use cases of these things 766 00:37:21,680 --> 00:37:23,760 are so specialized and unique that you kind of need 767 00:37:23,760 --> 00:37:26,480 to build out some internal expertise. 768 00:37:26,480 --> 00:37:30,240 And everyone's just kind of figuring that out now. 769 00:37:30,240 --> 00:37:34,720 So I guess I'd expect all of this to get better. 770 00:37:37,440 --> 00:37:42,800 But yeah, I guess I can't offer a win as soon as possible. 771 00:37:42,800 --> 00:37:46,080 It's definitely what we're working on. 772 00:37:46,080 --> 00:37:48,240 But it's clear this is not going anywhere. 773 00:37:48,240 --> 00:37:50,200 And there's a ton of potential. 774 00:37:50,200 --> 00:37:55,040 Like, I'm delighted by just like chat GPT on a daily basis 775 00:37:55,040 --> 00:37:58,160 and thinking of ideas for how this could be applied 776 00:37:58,160 --> 00:38:02,680 to different processes within organizations. 777 00:38:02,680 --> 00:38:03,920 Yeah, 100%. 778 00:38:03,920 --> 00:38:06,600 It's a really good brainstorm partner. 779 00:38:06,600 --> 00:38:07,760 You could give it some ideas. 780 00:38:07,760 --> 00:38:09,400 It could really, really helps out. 781 00:38:09,400 --> 00:38:11,200 And you can have a nice little back and forth. 782 00:38:11,200 --> 00:38:14,800 It generates very interesting ideas. 783 00:38:14,800 --> 00:38:17,720 And then you were touching upon another interesting thing, 784 00:38:17,720 --> 00:38:23,160 which was like the hardware that's involved with these systems. 785 00:38:23,160 --> 00:38:27,000 And obviously, there's an NVIDIA, which is a huge player. 786 00:38:27,000 --> 00:38:29,520 And then Google has their TPUs. 787 00:38:29,520 --> 00:38:32,440 And then there's this new thing like LPU. 788 00:38:32,440 --> 00:38:36,200 It's very interesting to think that now there's hardware 789 00:38:36,200 --> 00:38:41,040 that's going to be designed specifically for these use cases. 790 00:38:41,040 --> 00:38:43,080 So yeah, it'll be interesting to see 791 00:38:43,080 --> 00:38:45,840 can we get whoever, those companies, 792 00:38:45,840 --> 00:38:51,200 get the latency down to a point where 793 00:38:51,200 --> 00:38:53,840 you can actually make an API call, let's say. 794 00:38:53,840 --> 00:38:57,160 I guess there'll still be some challenges there, 795 00:38:57,160 --> 00:39:00,440 no matter what, as long as there's an API call involved. 796 00:39:00,440 --> 00:39:03,840 But if you're doing it locally, you also 797 00:39:03,840 --> 00:39:05,880 made another really good point. 798 00:39:05,880 --> 00:39:10,240 I think people tend to, it's like a new idea, 799 00:39:10,240 --> 00:39:12,240 like a maximum viable product. 800 00:39:12,240 --> 00:39:16,560 They'll use chat GPT to get a really good version of something. 801 00:39:16,560 --> 00:39:18,960 Then thinking, oh, then when we scale, 802 00:39:18,960 --> 00:39:23,960 we'll substitute it for Maestro or Llama or some other model. 803 00:39:23,960 --> 00:39:25,280 But it's not that simple. 804 00:39:25,280 --> 00:39:31,680 It's not really as simple as a plug and play. 805 00:39:31,680 --> 00:39:35,800 Yeah, so I guess along the same vein, 806 00:39:35,800 --> 00:39:38,200 what's an important question that you believe 807 00:39:38,200 --> 00:39:42,640 remains unanswered in machine learning? 808 00:39:42,640 --> 00:39:46,160 We've been in the space long enough to see what's happening 809 00:39:46,160 --> 00:39:46,920 here. 810 00:39:46,920 --> 00:39:51,800 We played with GPT-2, we played with GPT-3. 811 00:39:51,800 --> 00:39:55,480 We thought these were cool. 812 00:39:55,480 --> 00:39:58,000 We were telling our friends and family about it 813 00:39:58,000 --> 00:39:59,840 and having them try it. 814 00:39:59,840 --> 00:40:02,560 Right. 815 00:40:02,560 --> 00:40:06,680 It wasn't until the really instruction fine-tuned and chat 816 00:40:06,680 --> 00:40:08,880 GPT-stick stuff came out where it was like, 817 00:40:08,880 --> 00:40:10,200 whoa, this is really cool. 818 00:40:10,200 --> 00:40:16,880 But also, the models had gotten better at that point. 819 00:40:16,880 --> 00:40:24,760 So you just plot that stuff out on a graph. 820 00:40:24,760 --> 00:40:28,400 Like year thing was made and how good it was. 821 00:40:28,400 --> 00:40:32,680 Like the main limiting factor is the speed and cost 822 00:40:32,680 --> 00:40:35,040 of the chips running these things. 823 00:40:35,040 --> 00:40:37,120 And all indications are they get better 824 00:40:37,120 --> 00:40:41,920 if we're able to throw more computing power at them. 825 00:40:41,920 --> 00:40:44,440 So it's a weighted game. 826 00:40:44,440 --> 00:40:47,800 We're just waiting, essentially, for Moore's law, 827 00:40:47,800 --> 00:40:53,120 which happens to be an exponentially increasing 828 00:40:53,120 --> 00:40:57,360 phenomenon for these models to get better. 829 00:40:57,360 --> 00:41:02,160 So the question to me is, all right, well, when does that 830 00:41:02,160 --> 00:41:03,560 just mean we get AGI? 831 00:41:03,560 --> 00:41:06,480 I mean, this is a big question for open AI. 832 00:41:06,480 --> 00:41:08,760 Can we just continue to scale this thing up? 833 00:41:08,760 --> 00:41:11,720 And we have a model that's generally, however we 834 00:41:11,720 --> 00:41:21,080 want to define generally more capable than humanity. 835 00:41:21,080 --> 00:41:22,920 That's a big unquestioned answer for me. 836 00:41:22,920 --> 00:41:27,320 It's something I think about a lot. 837 00:41:27,320 --> 00:41:30,480 I think what's been really interesting in terms 838 00:41:30,480 --> 00:41:33,040 of unanswered or what I think will probably 839 00:41:33,040 --> 00:41:34,800 be some of the most interesting stuff 840 00:41:34,800 --> 00:41:36,000 in the next couple of years. 841 00:41:36,000 --> 00:41:39,960 Is all the multimodal work that's happening. 842 00:41:39,960 --> 00:41:44,000 So Gemini released their million token contact length, 843 00:41:44,000 --> 00:41:46,200 which means now we can just throw videos in there. 844 00:41:46,200 --> 00:41:48,400 And the stuff you can do with video is pretty cool. 845 00:41:51,360 --> 00:41:53,880 Just in my own personal usage of chat GPT, 846 00:41:53,880 --> 00:41:56,120 the image stuff has been amazing. 847 00:41:56,120 --> 00:41:58,000 Like I can take a picture of something 848 00:41:58,000 --> 00:42:03,120 I need transcribed or translated or I 849 00:42:03,120 --> 00:42:06,480 want you to count calories in my refrigerator. 850 00:42:06,480 --> 00:42:09,760 Like it's very cool what you can do just by adding imagery. 851 00:42:09,760 --> 00:42:14,160 And then if we throw audio and video, the use cases, 852 00:42:14,160 --> 00:42:18,120 and then if we make it faster to get input and output 853 00:42:18,120 --> 00:42:22,720 into that thing, the use cases are boundless. 854 00:42:22,720 --> 00:42:26,960 So I think that's a long winded way of saying 855 00:42:26,960 --> 00:42:31,960 the main problem here is just like more compute that's cheaper. 856 00:42:31,960 --> 00:42:38,160 And this is why NVIDIA stock is going to the moon. 857 00:42:38,160 --> 00:42:38,920 Through the roof. 858 00:42:38,920 --> 00:42:40,840 Yeah, absolutely. 859 00:42:40,840 --> 00:42:42,320 And it's like I saw it too. 860 00:42:42,320 --> 00:42:45,400 It's like I knew it was going to happen. 861 00:42:45,400 --> 00:42:46,760 Should have gotten deeper into that. 862 00:42:46,760 --> 00:42:55,920 Anyway, speaking of AGI, I think everyone 863 00:42:55,920 --> 00:42:57,800 has a different definition for it. 864 00:42:57,800 --> 00:43:00,960 Like slightly, I think. 865 00:43:00,960 --> 00:43:03,600 Do you feel like you have a good definition for AGI? 866 00:43:03,600 --> 00:43:04,120 Or so? 867 00:43:04,120 --> 00:43:05,920 No, I don't have a good definition. 868 00:43:05,920 --> 00:43:09,360 Well, I want to solve real science. 869 00:43:09,360 --> 00:43:13,880 Like solve some hairy problems that our best scientists 870 00:43:13,880 --> 00:43:14,920 can't solve. 871 00:43:14,920 --> 00:43:17,800 Then it's like, all right. 872 00:43:17,800 --> 00:43:19,080 Right. 873 00:43:19,080 --> 00:43:21,240 It's achievement unlocked. 874 00:43:21,240 --> 00:43:22,240 It can do it. 875 00:43:22,240 --> 00:43:23,560 So that's like what? 876 00:43:23,560 --> 00:43:27,840 Some unsolved math problems, some new protein thing. 877 00:43:27,840 --> 00:43:28,360 Well, yeah. 878 00:43:28,360 --> 00:43:33,120 People, they recently had a model like solve a proof 879 00:43:33,120 --> 00:43:34,120 that none of us could solve. 880 00:43:34,120 --> 00:43:35,680 So maybe it's here. 881 00:43:35,680 --> 00:43:39,680 Yeah, but if you look into it, they had it do it like 1,000 882 00:43:39,680 --> 00:43:40,360 times. 883 00:43:40,360 --> 00:43:42,080 And then they had mathematicians review it. 884 00:43:42,080 --> 00:43:43,240 And they found like, oh, OK. 885 00:43:43,240 --> 00:43:45,600 A handful of times this actually worked. 886 00:43:45,600 --> 00:43:46,200 I think that. 887 00:43:46,200 --> 00:43:46,600 I don't know. 888 00:43:46,600 --> 00:43:48,400 That's what I was reading about. 889 00:43:48,400 --> 00:43:49,720 But yes, it's possible. 890 00:43:49,720 --> 00:43:50,640 It's possible. 891 00:43:50,640 --> 00:43:53,200 Now, I think it's really, yeah. 892 00:43:53,200 --> 00:43:55,800 I mean, a lot of people way smarter than me. 893 00:43:55,800 --> 00:43:58,400 I've spent a lot of time trying to define this. 894 00:43:58,400 --> 00:44:00,960 So I'm not going to even attempt it. 895 00:44:00,960 --> 00:44:06,400 But it's one of those things where you probably 896 00:44:06,400 --> 00:44:08,960 know it when you see it. 897 00:44:08,960 --> 00:44:09,920 I don't know. 898 00:44:09,920 --> 00:44:15,960 I think it's going to be remarkable and scary. 899 00:44:15,960 --> 00:44:21,080 But it seems like, I'll also say, 900 00:44:21,080 --> 00:44:25,360 there's a long history of the machine learning 901 00:44:25,360 --> 00:44:27,880 world kind of over-promising and under-delivering 902 00:44:27,880 --> 00:44:29,080 when it comes to this stuff. 903 00:44:29,080 --> 00:44:33,920 So I would not be surprised if it takes us longer 904 00:44:33,920 --> 00:44:37,360 than the next generation of GPT here. 905 00:44:37,360 --> 00:44:44,720 But I do think there's a reasonable likelihood 906 00:44:44,720 --> 00:44:50,080 that in my lifetime, I get to see this, which is awesome. 907 00:44:50,080 --> 00:44:51,360 Scary. 908 00:44:51,360 --> 00:44:52,520 But I mean, like, wow. 909 00:44:52,520 --> 00:44:57,720 Like, I managed to be put on this Earth during a time 910 00:44:57,720 --> 00:45:03,960 when this evolved ape created this other thing that somehow 911 00:45:03,960 --> 00:45:04,480 surpassed. 912 00:45:04,480 --> 00:45:09,320 But it's just a very special time to be alive 913 00:45:09,320 --> 00:45:11,760 and to have the privilege to be a part of the space 914 00:45:11,760 --> 00:45:15,280 and kind of see it happen is pretty remarkable. 915 00:45:15,280 --> 00:45:16,560 Yeah, absolutely. 916 00:45:16,560 --> 00:45:22,240 It's like the most exciting time to be in machine learning. 917 00:45:22,240 --> 00:45:24,960 Changing gears a tiny bit. 918 00:45:24,960 --> 00:45:31,120 So you've been involved in two successful machine learning 919 00:45:31,120 --> 00:45:32,480 companies. 920 00:45:32,480 --> 00:45:37,080 What does it take to sort of take part 921 00:45:37,080 --> 00:45:39,640 in something like entrepreneurship in a field 922 00:45:39,640 --> 00:45:42,600 like machine learning where there's so much uncertainty? 923 00:45:42,600 --> 00:45:45,080 What are some of the lessons that you've learned? 924 00:45:45,080 --> 00:45:50,040 Well, I think lesson number one, you 925 00:45:50,040 --> 00:45:54,440 have to love what you're doing. 926 00:45:54,440 --> 00:45:56,440 And specifically with a start, it's like, well, 927 00:45:56,440 --> 00:46:02,520 you need to love the people that you're selling software to, 928 00:46:02,520 --> 00:46:06,880 the people you're solving problems for. 929 00:46:06,880 --> 00:46:14,680 And for me, machine learning, the intelligence, 930 00:46:14,680 --> 00:46:16,920 the thoughtfulness, the kinds of problems 931 00:46:16,920 --> 00:46:20,920 that can be solved with it just made it something 932 00:46:20,920 --> 00:46:22,720 that I could get very passionate about and put 933 00:46:22,720 --> 00:46:24,800 a ton of energy into. 934 00:46:24,800 --> 00:46:29,160 There's a lot of no one cares, especially in the beginning. 935 00:46:29,160 --> 00:46:30,280 Like you're building this thing. 936 00:46:30,280 --> 00:46:31,000 You think it's cool. 937 00:46:31,000 --> 00:46:31,760 You care a lot. 938 00:46:31,760 --> 00:46:32,400 You go out. 939 00:46:32,400 --> 00:46:33,480 You share it with people. 940 00:46:33,480 --> 00:46:40,000 And most people really do not care. 941 00:46:40,000 --> 00:46:45,640 So you need to have grit to push through that, 942 00:46:45,640 --> 00:46:51,880 to stay positive, to continue putting one foot in front 943 00:46:51,880 --> 00:46:53,480 of the other every day. 944 00:46:53,480 --> 00:46:58,640 I think others have given that advice just around persistence 945 00:46:58,640 --> 00:47:02,520 and being able to keep trying. 946 00:47:05,720 --> 00:47:08,840 But yeah, I guess for me, it's just like the main thing 947 00:47:08,840 --> 00:47:11,600 is you can go to a conference with your users 948 00:47:11,600 --> 00:47:12,760 and be energized. 949 00:47:12,760 --> 00:47:16,480 That would be the main piece of advice. 950 00:47:16,480 --> 00:47:18,640 Because if you don't have that, it's 951 00:47:18,640 --> 00:47:25,680 going to be really hard to keep going when you haven't necessarily 952 00:47:25,680 --> 00:47:29,800 found that product market fit or success in the space. 953 00:47:29,800 --> 00:47:30,840 Right. 954 00:47:30,840 --> 00:47:36,320 How did you know when you hit product market fit? 955 00:47:36,320 --> 00:47:37,240 Is it a feeling? 956 00:47:37,240 --> 00:47:41,080 Is it was there something that clicked where you had it, 957 00:47:41,080 --> 00:47:43,640 or was just about having a certain number of users, 958 00:47:43,640 --> 00:47:45,960 certain value that users were getting? 959 00:47:45,960 --> 00:47:48,000 I feel like that's something that's very hard. 960 00:47:48,000 --> 00:47:50,680 Like a lot of startup struggle with understanding, 961 00:47:50,680 --> 00:47:54,760 like have I reached product market fit? 962 00:47:54,760 --> 00:47:55,880 Yeah. 963 00:47:55,880 --> 00:47:58,560 Well, there's like first, just getting users. 964 00:47:58,560 --> 00:48:01,840 So that's big. 965 00:48:01,840 --> 00:48:05,960 But there's a lot of things you could do on the internet, 966 00:48:05,960 --> 00:48:10,480 especially if you have millions of VC dollars that give you 967 00:48:10,480 --> 00:48:12,920 a bunch of users that aren't necessarily 968 00:48:12,920 --> 00:48:16,480 ones that will stick around or be all that valuable. 969 00:48:16,480 --> 00:48:17,120 Right. 970 00:48:17,120 --> 00:48:24,360 And Lucas and I have always approached entrepreneurship 971 00:48:24,360 --> 00:48:27,920 like as a small business that really 972 00:48:27,920 --> 00:48:32,600 needs to earn every dollar and just make it work. 973 00:48:32,600 --> 00:48:37,840 So early on for us, it was those initial conversations 974 00:48:37,840 --> 00:48:40,840 with your very first customers where you're going to go, 975 00:48:40,840 --> 00:48:44,520 all right, we want to charge you for this software. 976 00:48:44,520 --> 00:48:47,160 You've got to come up with a price. 977 00:48:47,160 --> 00:48:51,080 It's kind of a harrowing process. 978 00:48:51,080 --> 00:48:53,920 But then to see customers actually say, yes, 979 00:48:53,920 --> 00:48:55,120 we want to pay you this. 980 00:48:55,120 --> 00:48:56,520 This is valuable. 981 00:48:56,520 --> 00:48:59,720 And seeing them continue to engage with the product. 982 00:48:59,720 --> 00:49:03,760 And it was probably like after a year of having paying 983 00:49:03,760 --> 00:49:06,680 customers and seeing that they actually renewed. 984 00:49:06,680 --> 00:49:10,400 All right, well, there's clearly something here. 985 00:49:10,400 --> 00:49:13,200 But even after getting those first couple of customers, 986 00:49:13,200 --> 00:49:16,360 it's like, we spent a lot of time with them. 987 00:49:16,360 --> 00:49:18,040 We held their hands a ton. 988 00:49:18,040 --> 00:49:19,120 Is this scalable? 989 00:49:19,120 --> 00:49:22,480 Are we going to be able to find broader market fit here? 990 00:49:22,480 --> 00:49:25,280 There's a lot of doubt in those early days. 991 00:49:25,280 --> 00:49:26,480 Right. 992 00:49:26,480 --> 00:49:28,040 Yeah. 993 00:49:28,040 --> 00:49:30,920 Yeah, so I guess it's not just about users. 994 00:49:30,920 --> 00:49:36,040 If you're creating a software product that anyone can use. 995 00:49:36,040 --> 00:49:39,920 Because users can be, you can do anything, anyone 996 00:49:39,920 --> 00:49:44,840 that's seen Silicon Valley. 997 00:49:44,840 --> 00:49:46,240 Have you watched Silicon Valley? 998 00:49:46,240 --> 00:49:47,240 Mm-hmm. 999 00:49:47,240 --> 00:49:49,840 Yeah. 1000 00:49:49,840 --> 00:49:51,800 But it's not just about getting users. 1001 00:49:51,800 --> 00:49:53,760 It's about retention. 1002 00:49:53,760 --> 00:49:55,840 And actually have them continue to use it. 1003 00:49:55,840 --> 00:49:59,360 And being able to continue to see how they're using it. 1004 00:49:59,360 --> 00:50:02,440 And yeah, pricing is always very tricky. 1005 00:50:02,440 --> 00:50:06,640 Because it can't just be like, however much they're willing 1006 00:50:06,640 --> 00:50:10,960 to pay, you actually have to equate that value to something. 1007 00:50:10,960 --> 00:50:13,120 So yeah, that must be very tricky. 1008 00:50:13,120 --> 00:50:14,120 Any other lessons from that? 1009 00:50:14,120 --> 00:50:17,440 Well, in the beginning, though, it is kind of an exercise of like, 1010 00:50:17,440 --> 00:50:19,680 how much do you want to pay? 1011 00:50:19,680 --> 00:50:20,160 Right. 1012 00:50:20,160 --> 00:50:22,000 I mean, you're trying to price this product that 1013 00:50:22,000 --> 00:50:27,160 has no precedent in the market. 1014 00:50:27,160 --> 00:50:29,040 Yeah, it's wild. 1015 00:50:29,040 --> 00:50:29,600 But it is. 1016 00:50:29,600 --> 00:50:33,200 You're kind of pulling numbers out of a hat. 1017 00:50:33,200 --> 00:50:35,160 Right. 1018 00:50:35,160 --> 00:50:36,560 I see the other piece on users. 1019 00:50:36,560 --> 00:50:40,080 Like an example, with both Waste and Biasis and CrowdFlight 1020 00:50:40,080 --> 00:50:46,320 Figure 8, we engaged a lot with the academic community. 1021 00:50:46,320 --> 00:50:50,600 And you're not monetizing that community. 1022 00:50:50,600 --> 00:50:53,200 There's like no, like you might be 1023 00:50:53,200 --> 00:50:56,200 able to get a university to pay a little bit for the software. 1024 00:50:56,200 --> 00:50:58,480 But the amount of work and pain you're 1025 00:50:58,480 --> 00:51:03,280 going to have to go through to get that done is a lot and not 1026 00:51:03,280 --> 00:51:04,400 worth it. 1027 00:51:04,400 --> 00:51:09,560 And then you might be able to get a handful of the academics 1028 00:51:09,560 --> 00:51:10,680 to pay for the software. 1029 00:51:10,680 --> 00:51:12,520 But the dollars are going to be really small. 1030 00:51:12,520 --> 00:51:17,080 And they have pretty tight budgets 1031 00:51:17,080 --> 00:51:22,040 and don't generally want to pay for software. 1032 00:51:22,040 --> 00:51:24,800 But we always would invest in that community 1033 00:51:24,800 --> 00:51:31,680 because we knew that if you're doing this work in academia, 1034 00:51:31,680 --> 00:51:34,280 eventually you're going to get a job in industry. 1035 00:51:34,280 --> 00:51:36,640 And you'll want to use the tools that 1036 00:51:36,640 --> 00:51:38,280 help you do your best work in academia 1037 00:51:38,280 --> 00:51:41,760 and hopefully bring us along. 1038 00:51:41,760 --> 00:51:44,440 But the end goal of the business is always 1039 00:51:44,440 --> 00:51:47,520 to close those larger deals with the various enterprises. 1040 00:51:47,520 --> 00:51:53,040 So you've got to be really smart about how you do that. 1041 00:51:53,040 --> 00:51:55,360 And there is some tension between, 1042 00:51:55,360 --> 00:51:58,160 all right, let's give as much of this away for free, 1043 00:51:58,160 --> 00:52:02,560 while also being able to monetize for industry. 1044 00:52:02,560 --> 00:52:05,200 Right, because the value that you get from people using 1045 00:52:05,200 --> 00:52:09,160 your software, figuring out what breaks, what doesn't break, 1046 00:52:09,160 --> 00:52:14,400 what people are getting value from, that's invaluable. 1047 00:52:14,400 --> 00:52:17,320 But you also don't want to just, you can't just give it away 1048 00:52:17,320 --> 00:52:18,160 forever. 1049 00:52:18,160 --> 00:52:20,280 At some point, it's a business. 1050 00:52:20,280 --> 00:52:24,080 There's a certain bottom line that you have to start collecting 1051 00:52:24,080 --> 00:52:26,640 some sort of fee. 1052 00:52:26,640 --> 00:52:27,680 But it's very interesting. 1053 00:52:27,680 --> 00:52:29,800 You mentioned in the beginning you 1054 00:52:29,800 --> 00:52:32,680 were doing things that were more almost consultative. 1055 00:52:32,680 --> 00:52:34,560 So when you were small, you were doing things 1056 00:52:34,560 --> 00:52:36,960 that didn't necessarily scale. 1057 00:52:36,960 --> 00:52:41,200 But did you know at the time that that was the case 1058 00:52:41,200 --> 00:52:44,760 and that in hopes that one day you'd be able to reach a point 1059 00:52:44,760 --> 00:52:47,800 where it would? 1060 00:52:47,800 --> 00:52:48,960 I mean, in the beginning, it's just 1061 00:52:48,960 --> 00:52:51,520 like you're trying to get anyone who will engage 1062 00:52:51,520 --> 00:52:52,760 to continue engaging. 1063 00:52:52,760 --> 00:52:54,120 So that was priceless. 1064 00:52:54,120 --> 00:52:56,600 Like, yes, the founders will drive down 1065 00:52:56,600 --> 00:53:02,680 to Mountain View every week to meet with the team at Toyota. 1066 00:53:02,680 --> 00:53:06,560 That's invaluable. 1067 00:53:06,560 --> 00:53:08,920 Now, we can't keep doing that forever. 1068 00:53:08,920 --> 00:53:10,880 But it was right for us to do it. 1069 00:53:10,880 --> 00:53:13,040 And I think of it less as like consultative 1070 00:53:13,040 --> 00:53:17,840 is that's something as an entrepreneur you always 1071 00:53:17,840 --> 00:53:19,360 need to be really careful with. 1072 00:53:19,360 --> 00:53:21,760 Because you don't want to make a consulting company that's 1073 00:53:21,760 --> 00:53:25,800 building bespoke things for different people 1074 00:53:25,800 --> 00:53:28,800 where there isn't a central platform or service that 1075 00:53:28,800 --> 00:53:34,640 can have the benefits of scale across many, many, many 1076 00:53:34,640 --> 00:53:35,400 different customers. 1077 00:53:35,400 --> 00:53:39,960 So we were working very closely and addressing 1078 00:53:39,960 --> 00:53:41,360 specific problems that they were having, 1079 00:53:41,360 --> 00:53:43,280 but always stepping back and saying, like, hey, 1080 00:53:43,280 --> 00:53:44,760 is this generally useful? 1081 00:53:44,760 --> 00:53:47,880 Will this also be something that someone working 1082 00:53:47,880 --> 00:53:50,320 in this other space could benefit from when 1083 00:53:50,320 --> 00:53:52,720 deciding whether or not we actually productized it 1084 00:53:52,720 --> 00:53:56,280 and put it into the product? 1085 00:53:56,280 --> 00:53:58,680 At my previous company, CrowdFlight Figure 8, 1086 00:53:58,680 --> 00:54:02,440 that was helping customers generate labeled data sets 1087 00:54:02,440 --> 00:54:04,840 for their machine learning model efforts. 1088 00:54:04,840 --> 00:54:07,800 That would often turn into actual consulting, 1089 00:54:07,800 --> 00:54:09,560 which was really hard. 1090 00:54:09,560 --> 00:54:11,320 Like, we're using our own software 1091 00:54:11,320 --> 00:54:13,760 on behalf of the customer, or we're 1092 00:54:13,760 --> 00:54:15,680 going deep into their specific use 1093 00:54:15,680 --> 00:54:17,600 cakes and helping them design. 1094 00:54:17,600 --> 00:54:22,760 And that makes for a very different business dynamic 1095 00:54:22,760 --> 00:54:25,520 than just selling a software license. 1096 00:54:25,520 --> 00:54:29,840 Yeah, I think that's because getting annotated data 1097 00:54:29,840 --> 00:54:32,360 is so much harder than people think it is. 1098 00:54:32,360 --> 00:54:36,560 Because it's not just like, oh, get good data. 1099 00:54:36,560 --> 00:54:39,280 Like what you were saying earlier, like, what does good mean? 1100 00:54:39,280 --> 00:54:41,800 Right, you need to create a set of annotation instructions. 1101 00:54:41,800 --> 00:54:43,920 You need to create the tooling around it. 1102 00:54:43,920 --> 00:54:45,760 And you actually have to know somehow 1103 00:54:45,760 --> 00:54:47,320 if you're collecting it. 1104 00:54:47,320 --> 00:54:51,800 And then often, this task, it won't even be objective. 1105 00:54:51,800 --> 00:54:53,520 There'll be some subjective nature to it, 1106 00:54:53,520 --> 00:54:56,880 and there'll be this low inter-annotator agreement. 1107 00:54:56,880 --> 00:54:59,320 So how do you even measure if you're getting good data? 1108 00:54:59,320 --> 00:55:02,520 So I'm sure there were so many challenges there. 1109 00:55:02,520 --> 00:55:04,800 But yet, such an important problem, 1110 00:55:04,800 --> 00:55:08,040 such an important thing to try to solve. 1111 00:55:08,040 --> 00:55:11,720 And way ahead of the game. 1112 00:55:11,720 --> 00:55:15,080 Like that was back in 2007, 2008. 1113 00:55:15,080 --> 00:55:19,000 I mean, thinking about the data-centric movement that's 1114 00:55:19,000 --> 00:55:20,640 taken place over the last few years, 1115 00:55:20,640 --> 00:55:24,960 like you knew that a long time ago. 1116 00:55:24,960 --> 00:55:27,440 Yeah, we were definitely too early to market 1117 00:55:27,440 --> 00:55:28,600 with that first company. 1118 00:55:28,600 --> 00:55:33,720 But we learned a ton and got to work 1119 00:55:33,720 --> 00:55:36,800 with a ton of really impressive machine learning teams 1120 00:55:36,800 --> 00:55:37,640 over the years. 1121 00:55:37,640 --> 00:55:39,800 I wouldn't take it back. 1122 00:55:39,800 --> 00:55:40,320 That's good. 1123 00:55:40,320 --> 00:55:43,000 Yeah, I mean, you get to learn about some of the problems. 1124 00:55:43,000 --> 00:55:48,080 I see how well scale has done, which started 10 years 1125 00:55:48,080 --> 00:55:51,040 after we started and think like, oh, if we had just 1126 00:55:51,040 --> 00:55:54,120 timed our go-to-market a little differently, 1127 00:55:54,120 --> 00:55:56,040 but now they're awesome. 1128 00:55:56,040 --> 00:55:58,000 They've executed it amazingly. 1129 00:55:58,000 --> 00:56:03,240 Yeah, the timing of things, there is a certain, 1130 00:56:03,240 --> 00:56:05,160 I never like using the term luck, 1131 00:56:05,160 --> 00:56:07,680 but there is a certain luck to timing, 1132 00:56:07,680 --> 00:56:09,360 especially for entrepreneurship. 1133 00:56:09,360 --> 00:56:12,960 You have to be excited in developing this thing 1134 00:56:12,960 --> 00:56:17,040 at the right time when other people are, 1135 00:56:17,040 --> 00:56:20,120 where some amount of people are ready for it at least. 1136 00:56:20,120 --> 00:56:23,000 You need to have some customer base. 1137 00:56:23,000 --> 00:56:24,960 I think that when it comes to creating 1138 00:56:24,960 --> 00:56:30,120 SaaS and a tech company, you have a team filled 1139 00:56:30,120 --> 00:56:32,440 with forward thinkers. 1140 00:56:32,440 --> 00:56:36,960 And that's not necessarily who the buyer is at a company. 1141 00:56:36,960 --> 00:56:40,240 It might not necessarily be the most forward thinker. 1142 00:56:40,240 --> 00:56:43,080 They might be a little bit more on the conservative side, 1143 00:56:43,080 --> 00:56:45,200 not willing to take certain risks. 1144 00:56:45,200 --> 00:56:47,800 And then you have to try to show them value, 1145 00:56:47,800 --> 00:56:49,120 which can be really tough. 1146 00:56:52,160 --> 00:56:54,280 Yeah, just thinking about some things 1147 00:56:54,280 --> 00:56:55,840 and the challenges of entrepreneurship. 1148 00:56:55,840 --> 00:56:57,600 But also, that's what makes it fun. 1149 00:56:57,600 --> 00:56:59,320 And then you combine it with machine learning. 1150 00:56:59,320 --> 00:57:00,720 It makes it even more fun. 1151 00:57:00,720 --> 00:57:02,040 There you go. 1152 00:57:02,040 --> 00:57:03,280 Yeah. 1153 00:57:03,280 --> 00:57:09,600 So in your career, well, first off, 1154 00:57:09,600 --> 00:57:13,480 you've had some of the best titles, I have to say. 1155 00:57:13,480 --> 00:57:16,600 Chief Awesome Officer at one point. 1156 00:57:16,600 --> 00:57:19,400 Just your name or your initials at another point. 1157 00:57:19,400 --> 00:57:21,680 Pretty cool. 1158 00:57:21,680 --> 00:57:23,080 Any other really cool ones? 1159 00:57:25,560 --> 00:57:28,280 Yeah, I think Chief Awesome Officer, I just put on LinkedIn 1160 00:57:28,280 --> 00:57:28,800 for fun. 1161 00:57:28,800 --> 00:57:31,320 Oh, OK. 1162 00:57:31,320 --> 00:57:34,360 But CAO, it's got a nice ring to it. 1163 00:57:34,360 --> 00:57:35,600 It does have a nice ring to it. 1164 00:57:35,600 --> 00:57:39,840 CVP, that's my personal favorite because it's my initials. 1165 00:57:39,840 --> 00:57:41,680 It could also be corporate vice president. 1166 00:57:41,680 --> 00:57:44,160 Yeah. 1167 00:57:44,160 --> 00:57:48,000 Yeah, titles are there. 1168 00:57:48,000 --> 00:57:48,680 It's the title. 1169 00:57:48,680 --> 00:57:54,320 I suppose I like having a C title, but it doesn't. 1170 00:57:54,320 --> 00:57:57,800 My titles co-founder really at the end of the day. 1171 00:57:57,800 --> 00:58:00,720 And that's one of the things I love most about the job 1172 00:58:00,720 --> 00:58:07,320 is that I'll get kind of brought into anything at any time 1173 00:58:07,320 --> 00:58:13,600 and can be really versatile and just try to solve problems 1174 00:58:13,600 --> 00:58:15,960 pragmatically. 1175 00:58:15,960 --> 00:58:17,520 Yeah, that's what I was going to say. 1176 00:58:17,520 --> 00:58:20,840 It's just about solving problems so they 1177 00:58:20,840 --> 00:58:23,800 get to bring you in to solve problems. 1178 00:58:23,800 --> 00:58:24,880 Fixer. 1179 00:58:24,880 --> 00:58:25,640 You're the fixer. 1180 00:58:25,640 --> 00:58:32,360 The closer, the fixer, both of them, I'll give you this one. 1181 00:58:32,360 --> 00:58:37,240 What's one piece of advice that you would give yourself 1182 00:58:37,240 --> 00:58:40,440 or you wish you received 20 years ago, 15 years ago? 1183 00:58:40,440 --> 00:58:41,880 All right, this is great. 1184 00:58:41,880 --> 00:58:44,080 Yeah, yeah, yeah, yeah. 1185 00:58:44,080 --> 00:58:44,960 Find a hobby. 1186 00:58:48,240 --> 00:58:48,800 OK. 1187 00:58:48,800 --> 00:58:51,880 I think this is something I had, like other friends had told me, 1188 00:58:51,880 --> 00:58:53,800 like, yeah, I should do this. 1189 00:58:53,800 --> 00:58:55,840 Especially as an entrepreneur, it's always just like, 1190 00:58:55,840 --> 00:58:57,240 oh, there's not a lot of time. 1191 00:58:57,240 --> 00:59:00,640 My hobby is this project. 1192 00:59:00,640 --> 00:59:04,240 And I've definitely found that there's 1193 00:59:04,240 --> 00:59:09,800 only so far that that goes before you're just kind of burnout 1194 00:59:09,800 --> 00:59:16,720 and now you're worse off than if you had just spent 1195 00:59:16,720 --> 00:59:21,400 your 10, 20 hours of free time last week doing something else 1196 00:59:21,400 --> 00:59:25,200 that you're interested in or excited about. 1197 00:59:25,200 --> 00:59:28,080 So any hobbies that you want to share? 1198 00:59:28,080 --> 00:59:29,720 Have something exciting, interesting on the side. 1199 00:59:32,880 --> 00:59:38,680 The sad part is I still don't have a great hobby. 1200 00:59:38,680 --> 00:59:41,240 So you wish that somebody gave you that advice, I guess. 1201 00:59:41,240 --> 00:59:42,440 Yeah, exactly. 1202 00:59:42,440 --> 00:59:45,400 That's like legit advice, yeah. 1203 00:59:45,400 --> 00:59:47,680 I mean, the regulars, I enjoy reading. 1204 00:59:47,680 --> 00:59:50,920 I enjoy long walks, traveling. 1205 00:59:50,920 --> 00:59:54,680 But I don't think any of those quite qualify as a hobby. 1206 00:59:54,680 --> 00:59:57,960 I'm thinking I should go to the clay studio 1207 00:59:57,960 --> 01:00:02,920 and throw some clay or go weld some metal together or something. 1208 01:00:02,920 --> 01:00:04,280 But yeah. 1209 01:00:04,280 --> 01:00:08,560 I was going to say something maybe in the art realm. 1210 01:00:08,560 --> 01:00:10,000 Yeah. 1211 01:00:10,000 --> 01:00:14,080 OK, the final and the juiciest of questions. 1212 01:00:14,080 --> 01:00:18,200 What has a career in machine learning and entrepreneurship 1213 01:00:18,200 --> 01:00:21,560 taught you about life? 1214 01:00:21,560 --> 01:00:22,240 Oh, man. 1215 01:00:26,200 --> 01:00:29,840 Well, I'd say the entrepreneurship part has taught me 1216 01:00:29,840 --> 01:00:38,360 that there's the business, there's this idea, the customer. 1217 01:00:38,360 --> 01:00:42,520 All of these things we think about when 1218 01:00:42,520 --> 01:00:43,920 we think of the kinds of problems you're 1219 01:00:43,920 --> 01:00:46,800 going to have to deal with within a company. 1220 01:00:46,800 --> 01:00:50,680 The thing that I never thought about that much, 1221 01:00:50,680 --> 01:00:54,640 but is actually what I found to be the most important, 1222 01:00:54,640 --> 01:01:01,280 is the people within the company that you're creating. 1223 01:01:01,280 --> 01:01:04,400 You're hiring a bunch of folks to work on a problem. 1224 01:01:07,920 --> 01:01:11,960 But each of those individuals is another person 1225 01:01:11,960 --> 01:01:14,920 with their own problems, own stuff going on. 1226 01:01:14,920 --> 01:01:17,800 And the only way the organization is going to be effective 1227 01:01:17,800 --> 01:01:23,240 is if the people within it feel respected and treated 1228 01:01:23,240 --> 01:01:24,600 as humans with dignity. 1229 01:01:24,600 --> 01:01:30,920 And there's not some magic formula. 1230 01:01:30,920 --> 01:01:33,320 But this is what you do, such that everyone in your company 1231 01:01:33,320 --> 01:01:36,440 will now be seen as their full and true self. 1232 01:01:36,440 --> 01:01:39,360 But I think it's something important, especially 1233 01:01:39,360 --> 01:01:41,920 as an entrepreneur, as a leader in the company to think about 1234 01:01:41,920 --> 01:01:47,040 and to try to engage with as many people in the organization 1235 01:01:47,040 --> 01:01:52,720 as human beings as possible. 1236 01:01:52,720 --> 01:01:54,320 That's definitely a lesson. 1237 01:01:54,320 --> 01:01:58,760 I think the other piece that I've 1238 01:01:58,760 --> 01:02:00,840 learned in doing this over the years 1239 01:02:00,840 --> 01:02:06,280 is that I could still find that joy, that happiness of imagining 1240 01:02:06,280 --> 01:02:11,280 how to solve a problem and going out and solving it. 1241 01:02:11,280 --> 01:02:16,120 That being a creator, that's one of the aspects 1242 01:02:16,120 --> 01:02:20,040 of entrepreneurship that I love the most. 1243 01:02:20,040 --> 01:02:23,040 And it's been just amazing, even over the last six months, 1244 01:02:23,040 --> 01:02:27,880 to go and experiment with these new language models 1245 01:02:27,880 --> 01:02:31,360 and see what kind of side projects and tools 1246 01:02:31,360 --> 01:02:32,520 that I can create. 1247 01:02:32,520 --> 01:02:36,120 And I still have that same joy I had as a teenager when I made 1248 01:02:36,120 --> 01:02:41,520 my first website, and that's been awesome. 1249 01:02:41,520 --> 01:02:44,960 And just continuing to learn and to build. 1250 01:02:44,960 --> 01:02:46,000 That's awesome. 1251 01:02:46,000 --> 01:02:46,680 I love it. 1252 01:02:46,680 --> 01:02:49,280 I love it. 1253 01:02:49,280 --> 01:02:52,320 For people that are interested in learning more about you 1254 01:02:52,320 --> 01:02:54,720 or some of the work that you're doing at Weights and Biases, 1255 01:02:54,720 --> 01:02:58,480 where would you direct them? 1256 01:02:58,480 --> 01:03:04,560 Well, WB.com has information about the product itself 1257 01:03:04,560 --> 01:03:06,120 in the company. 1258 01:03:06,120 --> 01:03:10,440 There's also really cool links to different, what we call, 1259 01:03:10,440 --> 01:03:12,880 reports in the Weights and Biases platform, which 1260 01:03:12,880 --> 01:03:15,000 can be bits of research or analysis 1261 01:03:15,000 --> 01:03:18,520 or leveraging some of the new large language model stuff 1262 01:03:18,520 --> 01:03:19,800 we were talking about today. 1263 01:03:19,800 --> 01:03:21,760 That's really good content. 1264 01:03:21,760 --> 01:03:26,120 We have a YouTube channel, and we're on Twitter, LinkedIn. 1265 01:03:26,120 --> 01:03:30,920 Those are our primary social media outlets. 1266 01:03:30,920 --> 01:03:34,840 I'm on Twitter at VanPelt. 1267 01:03:34,840 --> 01:03:35,400 Ping me. 1268 01:03:35,400 --> 01:03:37,040 Hit me up. 1269 01:03:37,040 --> 01:03:38,440 It's been a pleasure. 1270 01:03:38,440 --> 01:03:40,440 Yes, it's been absolutely fantastic. 1271 01:03:40,440 --> 01:03:42,920 I really appreciate you giving me the time. 1272 01:03:42,920 --> 01:03:44,800 Thank you so much for the incredible work 1273 01:03:44,800 --> 01:03:47,280 that you're doing at Weights and Biases. 1274 01:03:47,280 --> 01:03:50,000 Thanks for letting me pick your brain for a little bit. 1275 01:03:50,000 --> 01:03:50,360 You bet. 1276 01:03:50,360 --> 01:03:51,000 This is fun. 1277 01:03:51,000 --> 01:03:52,000 Thanks for having me. 1278 01:03:52,000 --> 01:03:57,200 Thank you for tuning in to Learning from Machine Learning. 1279 01:03:57,200 --> 01:03:59,920 On this episode, we delved into the experiences 1280 01:03:59,920 --> 01:04:04,160 of Chris Van Pelt, co-founder of Weights and Biases, 1281 01:04:04,160 --> 01:04:06,840 gaining valuable insights into the current landscape 1282 01:04:06,840 --> 01:04:08,080 of the industry. 1283 01:04:08,080 --> 01:04:11,320 Chris explained the pivotal role of Weights and Biases 1284 01:04:11,320 --> 01:04:14,720 as a powerful developer tool, enabling ML engineers 1285 01:04:14,720 --> 01:04:18,280 to navigate through the complexities of experimentation, 1286 01:04:18,280 --> 01:04:22,040 data visualization, and model improvement. 1287 01:04:22,040 --> 01:04:23,680 His candid reflections on the challenges 1288 01:04:23,680 --> 01:04:27,120 in evaluating ML models and addressing the gap between AI 1289 01:04:27,120 --> 01:04:30,240 hype and reality offered a profound understanding 1290 01:04:30,240 --> 01:04:32,120 of the field's intricacies. 1291 01:04:32,120 --> 01:04:34,680 Drawing from his entrepreneurial experiences, 1292 01:04:34,680 --> 01:04:37,120 co-founding two machine learning companies, 1293 01:04:37,120 --> 01:04:41,360 Chris leaves us with lessons in resilience, innovation, 1294 01:04:41,360 --> 01:04:44,240 and a deep appreciation for the human dimension 1295 01:04:44,240 --> 01:04:45,640 within the tech line. 1296 01:04:45,640 --> 01:04:48,360 Don't forget to subscribe and share this episode 1297 01:04:48,360 --> 01:04:50,320 with your friends and colleagues. 1298 01:04:50,320 --> 01:05:16,400 Until next time, keep on learning.