WEBVTT

00:00:00.000 --> 00:00:01.919
but look honestly if you zoom out even a little

00:00:01.919 --> 00:00:04.719
bit all this stuff is so underhyped like you

00:00:04.719 --> 00:00:06.360
can't you can't hype it enough right i mean it's

00:00:06.360 --> 00:00:08.939
just like it's just amazing it's incredible this

00:00:08.939 --> 00:00:10.560
technology works really well it affects so many

00:00:10.560 --> 00:00:12.640
industries people are like maybe disappointed

00:00:12.640 --> 00:00:15.140
that like it didn't solve every single one of

00:00:15.140 --> 00:00:17.420
their business problems like right now and it's

00:00:17.420 --> 00:00:21.079
like relax like obviously it will soon just wait

00:00:21.079 --> 00:00:28.570
wait a month you know How did the best machine

00:00:28.570 --> 00:00:30.710
learning practitioners get involved in the field?

00:00:31.589 --> 00:00:34.469
What challenges have they faced? What has helped

00:00:34.469 --> 00:00:38.810
them flourish? Let's ask them. Welcome to Learning

00:00:38.810 --> 00:00:42.289
from Machine Learning. I'm your host, Seth Levine.

00:00:44.229 --> 00:00:47.090
Hello and welcome to Learning from Machine Learning.

00:00:47.409 --> 00:00:49.770
On this episode, we have a very special guest,

00:00:50.090 --> 00:00:52.969
Lukas Biewald, the co -founder and CEO of Weights

00:00:52.969 --> 00:00:56.310
and Biases. the host of one of my favorite podcasts,

00:00:56.750 --> 00:01:00.770
Gradient Ascent, one of the earliest AI entrepreneurs

00:01:00.770 --> 00:01:05.290
starting CrowdFlower Figure 8 back in 2007. Lucas,

00:01:05.349 --> 00:01:06.870
it is such a pleasure to have you on the show.

00:01:06.950 --> 00:01:10.969
Thank you so much. Thanks for having me. We're

00:01:10.969 --> 00:01:14.250
going to get right into it. So what initially

00:01:14.250 --> 00:01:17.609
attracted you to machine learning? I was always

00:01:17.609 --> 00:01:20.769
interested in AI. generally, like, you know,

00:01:20.870 --> 00:01:22.769
even when I was a kid, I think I really liked

00:01:22.769 --> 00:01:25.709
to play games. And so I thought a lot, you know,

00:01:25.769 --> 00:01:28.650
try to program, you know, my own kind of like,

00:01:28.829 --> 00:01:31.269
rule based systems to, you know, to win it, like

00:01:31.269 --> 00:01:33.209
connect four and stuff like that. So I was thinking

00:01:33.209 --> 00:01:36.870
about that a lot. And, um, you know, my dad actually

00:01:36.870 --> 00:01:39.500
read me that book, uh, Gertlischer Bach. And

00:01:39.500 --> 00:01:41.719
I didn't understand much of it at all, but it

00:01:41.719 --> 00:01:44.019
has these great parts where there's these sort

00:01:44.019 --> 00:01:46.879
of stories. And so the stories are pretty compelling.

00:01:47.239 --> 00:01:50.519
And then I think with that kid mind, I was like,

00:01:50.519 --> 00:01:53.859
you know, AI does seem like humanity's last project

00:01:53.859 --> 00:01:56.000
and kind of the most interesting thing you could

00:01:56.000 --> 00:01:58.659
possibly work on. And then I went to Stanford

00:01:58.659 --> 00:02:01.760
and I... I actually reached out cold to Daphne

00:02:01.760 --> 00:02:03.840
Kohler, who was kind of a young professor there

00:02:03.840 --> 00:02:06.079
at the time, and asked her what she did. She's

00:02:06.079 --> 00:02:08.099
like, okay, take my class in machine learning.

00:02:08.319 --> 00:02:09.939
The class was so much fun because it was kind

00:02:09.939 --> 00:02:12.740
of the fun parts. And actually, the best part

00:02:12.740 --> 00:02:15.219
of the class was this kind of reinforcement learning

00:02:15.219 --> 00:02:18.400
piece where you train a fellow program through

00:02:18.400 --> 00:02:20.810
reinforcement learning to... to get good at Othello.

00:02:21.810 --> 00:02:23.050
And yeah, from that point I was hooked because

00:02:23.050 --> 00:02:24.990
that's actually, Othello's a really satisfying

00:02:24.990 --> 00:02:26.389
reinforcement learning project you could even

00:02:26.389 --> 00:02:28.830
do back then where, you know, I kind of watched

00:02:28.830 --> 00:02:32.810
that program, you know, over a few nights, basically

00:02:32.810 --> 00:02:35.550
go from terrible where, you know, it couldn't

00:02:35.550 --> 00:02:38.169
beat anyone to just like, you know, crushing

00:02:38.169 --> 00:02:40.789
me every single time. And actually it's funny,

00:02:40.830 --> 00:02:42.870
you know, I got way better at Othello playing

00:02:42.870 --> 00:02:47.110
that game. I've actually never lost at Othello

00:02:47.110 --> 00:02:50.389
since training that computer because the computer's

00:02:50.389 --> 00:02:52.770
kind of teaching me how to play um towards the

00:02:52.770 --> 00:02:54.830
end of it although i've never played like a i

00:02:54.830 --> 00:02:56.949
don't know professional or like you know skilled

00:02:56.949 --> 00:02:59.750
a fellow player but um but yeah i mean that that

00:02:59.750 --> 00:03:01.449
was like the point where i got i got like you

00:03:01.449 --> 00:03:03.590
know really really kind of hooked on it and then

00:03:03.590 --> 00:03:04.969
it kind of went through like a sad part so i

00:03:04.969 --> 00:03:07.150
you know i started doing research with um daphne

00:03:07.150 --> 00:03:08.810
i kind of thought okay maybe you know i want

00:03:08.810 --> 00:03:11.129
to be like a professor of this but you know this

00:03:11.129 --> 00:03:13.310
is back in like 2004 2005 there really was not

00:03:13.310 --> 00:03:16.280
a lot of stuff Um, working, I mean, Google was

00:03:16.280 --> 00:03:17.960
like working really well, but a lot of that was

00:03:17.960 --> 00:03:21.580
kind of like page rank and, um, you know, a lot

00:03:21.580 --> 00:03:24.080
of the, the energy was around sort of like, you

00:03:24.080 --> 00:03:25.919
know, ranking ads that didn't, you know, that

00:03:25.919 --> 00:03:28.479
didn't feel like that satisfying. And I interviewed

00:03:28.479 --> 00:03:30.759
like some hedge funds and, you know, I was like,

00:03:30.860 --> 00:03:32.580
okay, like, I don't know if that's really like,

00:03:32.819 --> 00:03:34.319
you know, what I wanted to dedicate, you know,

00:03:34.400 --> 00:03:36.180
my life to. And I was actually thinking like,

00:03:36.259 --> 00:03:38.919
okay, you know, label data is kind of driving

00:03:38.919 --> 00:03:40.759
all the research, which is why I got into like

00:03:40.759 --> 00:03:42.419
the data labeling business. So I thought, okay,

00:03:42.419 --> 00:03:44.159
you know, like if we could create more label

00:03:44.159 --> 00:03:47.240
data, we could create more interesting applications,

00:03:47.280 --> 00:03:49.159
which is kind of like what I always, you know,

00:03:49.259 --> 00:03:50.479
what's kind of the most fun part of machine learning.

00:03:50.620 --> 00:03:52.099
It's like, you know, really seeing what it can

00:03:52.099 --> 00:03:54.939
do. And man, I've like lost even what the question

00:03:54.939 --> 00:04:02.099
was, but I'm no, you know, it's perfect. So that

00:04:02.099 --> 00:04:06.699
leads you right into 2007. Founding your was

00:04:06.699 --> 00:04:08.800
that that was your first company that was founded,

00:04:08.879 --> 00:04:11.280
right? Yeah. You were way ahead of the game.

00:04:11.500 --> 00:04:13.610
You were way ahead of the game. Yeah, although

00:04:13.610 --> 00:04:15.750
you say I was one of the first AI entrepreneurs.

00:04:15.930 --> 00:04:17.149
I don't think that's actually true. I mean, there's

00:04:17.149 --> 00:04:20.810
been a long... People have been building AI companies

00:04:20.810 --> 00:04:24.370
since the 70s, 80s. So I think the definition

00:04:24.370 --> 00:04:27.350
of AI kind of changes. Once we get something

00:04:27.350 --> 00:04:29.829
working, we no longer maybe consider it AI all

00:04:29.829 --> 00:04:33.269
the time. Right. Yeah. But yeah, that was actually

00:04:33.269 --> 00:04:36.589
sort of like an AI winter. It was also sort of

00:04:36.589 --> 00:04:40.000
like an economic winter where there's... You

00:04:40.000 --> 00:04:42.100
know, like Y Combinator just started, but if

00:04:42.100 --> 00:04:43.939
you look at those early stuff, you know, those

00:04:43.939 --> 00:04:46.199
are kind of much weirder ideas. And now I raised

00:04:46.199 --> 00:04:48.160
my first round. It took a year to raise a round

00:04:48.160 --> 00:04:51.439
on, um, $2 million, you know, valuation and we

00:04:51.439 --> 00:04:53.519
had revenue. So it's really a different time.

00:04:53.540 --> 00:04:55.319
Like Facebook was just starting to take off.

00:04:55.660 --> 00:04:57.860
And, and, and so another thing that was funny

00:04:57.860 --> 00:04:59.860
back then was I, I was basically, I was building

00:04:59.860 --> 00:05:03.259
an AI labeling company, but I was kind of coached

00:05:03.259 --> 00:05:05.420
to take AI, you know, out of the pitch. Cause

00:05:05.420 --> 00:05:08.740
that. It was such an AI winner. Investors did

00:05:08.740 --> 00:05:10.680
not want to invest in anything related to AI.

00:05:10.740 --> 00:05:12.980
That just seemed like a bad science project.

00:05:14.379 --> 00:05:18.420
That was a real sign of the times. It's funny

00:05:18.420 --> 00:05:20.279
because what was happening actually behind the

00:05:20.279 --> 00:05:22.959
scenes is these companies were starting to find

00:05:22.959 --> 00:05:27.860
success. Our labeling product got some quick

00:05:27.860 --> 00:05:31.720
traction with actually eBay was our first really

00:05:31.720 --> 00:05:34.120
big customer and that really pulled us along.

00:05:34.610 --> 00:05:36.829
Yeah, you mentioned a couple of really interesting

00:05:36.829 --> 00:05:38.990
points that I've been talking about also recently,

00:05:39.029 --> 00:05:41.790
like the moving goalposts of AI, right? Because

00:05:41.790 --> 00:05:44.149
like once it works, it's it's just an algorithm.

00:05:44.149 --> 00:05:47.230
It's like, I always think about computer vision.

00:05:47.329 --> 00:05:49.730
And I think about like face detection and things

00:05:49.730 --> 00:05:51.230
like that. I mean, those are just algorithms.

00:05:51.269 --> 00:05:53.149
But if you thought, you know, whatever decades

00:05:53.149 --> 00:05:55.189
before, if there would be a thing in your pocket

00:05:55.189 --> 00:05:58.110
that can identify where your face was, right,

00:05:58.110 --> 00:06:00.910
that would be obviously magical. Now we just

00:06:00.910 --> 00:06:03.009
think about it like it's just a given. Like,

00:06:03.069 --> 00:06:05.029
of course, that's how That's how things work.

00:06:05.430 --> 00:06:10.310
So I guess the modern era of AI entrepreneurship,

00:06:10.490 --> 00:06:11.889
I guess what I was saying early to the game,

00:06:12.009 --> 00:06:13.470
there are still companies that are trying to

00:06:13.470 --> 00:06:15.769
solve those same data labeling problems that

00:06:15.769 --> 00:06:18.930
you were addressing in 2007. So you could really

00:06:18.930 --> 00:06:23.649
kind of understand you had some inkling of something

00:06:23.649 --> 00:06:25.550
amazing happening. I mean, think about it. That

00:06:25.550 --> 00:06:29.209
was nearly nearly 20 years ago. And we're still

00:06:29.209 --> 00:06:33.420
trying to solve these problems. Right. Yeah,

00:06:33.639 --> 00:06:36.540
I usually ask like was there a moment that convinced

00:06:36.540 --> 00:06:38.639
you that this was the field that you wanted to

00:06:38.639 --> 00:06:40.699
build your career in? But it sounds like it was

00:06:40.699 --> 00:06:43.040
pretty early on or was there something later

00:06:43.040 --> 00:06:44.819
that happens that where you were like, yeah This

00:06:44.819 --> 00:06:47.220
is definitely what I want to be doing No, I mean,

00:06:47.339 --> 00:06:48.920
I think it was really early like I mean like

00:06:48.920 --> 00:06:51.600
really like a little kid kind of thinking about

00:06:51.600 --> 00:06:54.000
AI I think I remember somebody's dad I think

00:06:54.000 --> 00:06:56.439
told me about like simulated annealing days to

00:06:56.439 --> 00:06:58.379
call it which is kind of like a gradient descent

00:06:58.600 --> 00:07:02.360
And he told me something was designed in that

00:07:02.360 --> 00:07:04.680
way where they had the computer try different

00:07:04.680 --> 00:07:08.339
weights and then gradually make the weights better.

00:07:08.579 --> 00:07:11.660
It really just captured my imagination. I was

00:07:11.660 --> 00:07:13.160
writing code. I was like, what if computers could

00:07:13.160 --> 00:07:16.100
write this code? When you're a kid, it's not

00:07:16.100 --> 00:07:19.439
obvious that facial recognition would be hard,

00:07:20.319 --> 00:07:22.319
but multiplying two huge numbers together would

00:07:22.319 --> 00:07:25.009
be easy. you kind of come to that, you know,

00:07:25.149 --> 00:07:27.850
over time. And so I think I had like a lot of

00:07:27.850 --> 00:07:30.129
optimism, which has sort of like slowly, you

00:07:30.129 --> 00:07:32.709
know, crushed out of me through like, you know,

00:07:32.850 --> 00:07:35.389
actually like trying to try to do it, you know,

00:07:35.569 --> 00:07:38.610
from like, you know, 2003 to maybe like 2016.

00:07:38.610 --> 00:07:41.490
Actually, it's funny. I was, I was a big Go player.

00:07:41.529 --> 00:07:43.430
Like I loved playing Go. I think one of the things

00:07:43.430 --> 00:07:46.730
I liked about it was that computers are so bad

00:07:46.730 --> 00:07:48.850
at it. And if you, you know, if you do play Go,

00:07:48.970 --> 00:07:51.680
you kind of... Understand why like it's it's

00:07:51.680 --> 00:07:53.699
like it's interesting. It's like it goes like

00:07:53.699 --> 00:07:56.339
really designed for our brains to do it Well

00:07:56.339 --> 00:07:59.819
in a super cool and satisfying way, but you know,

00:07:59.939 --> 00:08:03.300
I remember when I was running I was running crowdflower

00:08:03.300 --> 00:08:07.680
figure eight when alpha go be Lisa doll. Yeah

00:08:07.850 --> 00:08:10.149
I mean, because I'm like a huge Go fan. I was

00:08:10.149 --> 00:08:12.009
like, actually, you know, I was like watching

00:08:12.009 --> 00:08:13.889
those games live and like, you know, watching

00:08:13.889 --> 00:08:15.689
the commentary and really thinking about it.

00:08:15.689 --> 00:08:17.889
And it was like, I actually had not really paid

00:08:17.889 --> 00:08:19.769
attention to the advances in deep learning. Like

00:08:19.769 --> 00:08:21.970
at that point, if anyone is ever like, hey, we

00:08:21.970 --> 00:08:23.430
have a new algorithm that does things better.

00:08:23.970 --> 00:08:25.750
You just have a huge skepticism because people

00:08:25.750 --> 00:08:27.189
kept saying that, kept saying that it wasn't

00:08:27.189 --> 00:08:29.269
true, wasn't true, especially in like the entrepreneurial

00:08:29.269 --> 00:08:31.209
space. It's a good pitch, you know, but it's

00:08:31.209 --> 00:08:32.909
actually like generally not true. So I was like

00:08:32.909 --> 00:08:37.070
super skeptical that, you know, AlphaGo was going

00:08:37.070 --> 00:08:40.000
to beat. uh, least at all. And then, you know,

00:08:40.019 --> 00:08:42.460
when it did, I was just like, Oh my God, you

00:08:42.460 --> 00:08:45.039
know, like this is like, that's kind of what

00:08:45.039 --> 00:08:47.200
made me realize that, okay, I need to, um, you

00:08:47.200 --> 00:08:48.759
know, kind of get more technical again. And I

00:08:48.759 --> 00:08:50.220
was like, look, and that's like how I ended up,

00:08:50.220 --> 00:08:52.100
you know, doing an unpaid internship at, at,

00:08:52.100 --> 00:08:54.279
at opening up. This is like very brief and wasn't

00:08:54.279 --> 00:08:56.159
on their payroll or anything, but I was like,

00:08:56.340 --> 00:08:58.600
I just need to go somewhere where people are

00:08:58.600 --> 00:09:01.240
doing the latest stuff and, and work with them

00:09:01.240 --> 00:09:03.139
because I, you know, running a data label and

00:09:03.139 --> 00:09:05.940
company, you work with like AI customers, but

00:09:05.940 --> 00:09:08.700
you're not really doing. a lot of AI internally.

00:09:08.960 --> 00:09:10.860
So I kind of was, it's funny. I was like, you

00:09:10.860 --> 00:09:12.299
know, my thirties and kind of feeling bad about

00:09:12.299 --> 00:09:15.399
myself. Like I had just become this like totally

00:09:15.399 --> 00:09:17.460
like deeply non -technical person and just sort

00:09:17.460 --> 00:09:20.240
of like a mediocre, you know, like entrepreneur

00:09:20.240 --> 00:09:23.100
and like just didn't even really know like, you

00:09:23.100 --> 00:09:24.460
know, what was kind of going on in the space.

00:09:24.559 --> 00:09:26.440
And so I'm so happy that I did that. Cause I

00:09:26.440 --> 00:09:30.029
think that like led to me. Getting a lot more

00:09:30.029 --> 00:09:32.190
technical again and and like actually like getting

00:09:32.190 --> 00:09:35.330
to enjoy This new era where like all the AI stuff

00:09:35.330 --> 00:09:37.490
works and like things surprise you by how fast

00:09:37.490 --> 00:09:39.289
they come rather than like how slow they come

00:09:39.289 --> 00:09:42.580
Yeah Going back to the point that you were making

00:09:42.580 --> 00:09:45.539
earlier, it's like back in whatever say the mid

00:09:45.539 --> 00:09:49.000
2000s when you didn't want to say AI in your

00:09:49.000 --> 00:09:51.779
pitch, thinking about what that's like now, where

00:09:51.779 --> 00:09:55.019
every single entrepreneur, whether it is AI or

00:09:55.019 --> 00:09:57.100
it's not AI, they'll just say it, it's become

00:09:57.100 --> 00:10:00.639
more of like a marketing term. And then yeah,

00:10:00.639 --> 00:10:03.899
the other point that you're making with Go, that's

00:10:03.899 --> 00:10:06.419
like one of the main stories in the book Genius

00:10:06.419 --> 00:10:10.220
Makers. And they talk about that like move, move.

00:10:10.220 --> 00:10:14.620
Yeah, yeah. Yeah, where it was not intuitive

00:10:14.620 --> 00:10:17.840
for a human to make that move, but it ended up

00:10:17.840 --> 00:10:21.419
being the right move, which is just such an interesting

00:10:21.419 --> 00:10:24.279
idea with AI in general, just like the ability

00:10:24.279 --> 00:10:26.360
for it to pick up patterns, but it's more than

00:10:26.360 --> 00:10:28.080
just picking up patterns, it's picking up patterns

00:10:28.080 --> 00:10:31.100
that humans might not be able to detect, which,

00:10:31.740 --> 00:10:33.779
yeah, unless you're kind of really in it, it's

00:10:33.779 --> 00:10:37.360
hard to understand the power of it. And I'll

00:10:37.360 --> 00:10:39.159
tell you, for the true Go aficionado, which I'm

00:10:39.159 --> 00:10:40.840
not like a... I don't know. I don't play a lot

00:10:40.840 --> 00:10:43.019
of Go these days, but I am pretty interested

00:10:43.019 --> 00:10:45.120
in it. As a kid, I played it a ton of Go. And

00:10:45.120 --> 00:10:48.480
I would always think, OK, what would God do?

00:10:48.700 --> 00:10:50.179
You know what I mean? I wish I could know. Because

00:10:50.179 --> 00:10:52.820
it feels so weird. The kind of optimal first

00:10:52.820 --> 00:10:55.820
move is the 4 -4 point, or maybe the 4 -3 point,

00:10:55.919 --> 00:10:57.720
which is just not the center of the board. It's

00:10:57.720 --> 00:11:00.179
not the edge of the board. It's just like, OK.

00:11:00.340 --> 00:11:03.259
And then it's debated. Is 4 -4 better or 4 -3

00:11:03.259 --> 00:11:05.179
better? And all these openings, there's a lot

00:11:05.179 --> 00:11:07.929
of tradition and like, You know, you're kind

00:11:07.929 --> 00:11:10.250
of like, okay, how true is it? Like, and there

00:11:10.250 --> 00:11:12.210
are people who come up with these like wild styles

00:11:12.210 --> 00:11:14.389
and they'd like work pretty well. And I just

00:11:14.389 --> 00:11:15.889
always was wondering, like, could we find out

00:11:15.889 --> 00:11:18.090
in my lifetime, like what the best, you know,

00:11:18.210 --> 00:11:20.649
moves are. And then when they trained it, like

00:11:20.649 --> 00:11:22.610
the, the alpha zero thing where they trained

00:11:22.610 --> 00:11:24.669
it with like not unprofessional games. I think

00:11:24.669 --> 00:11:26.710
the one that beat least at all was mostly trained

00:11:26.710 --> 00:11:28.330
on professional games and a little bit of self

00:11:28.330 --> 00:11:30.149
play to get a little bit better, but it kind

00:11:30.149 --> 00:11:33.379
of had a more. like a style that like came from

00:11:33.379 --> 00:11:35.980
humans, but then the ones trained without any

00:11:35.980 --> 00:11:38.860
human input are so interesting, both where they

00:11:38.860 --> 00:11:42.139
find like, you know, kind of famous human openings

00:11:42.139 --> 00:11:45.240
and like those, but also there were a few things

00:11:45.240 --> 00:11:47.759
that were missed, like really like obvious things

00:11:47.759 --> 00:11:52.279
that you could go back, you know, 20 years and

00:11:52.279 --> 00:11:55.159
actually explain them to like any Go player.

00:11:55.519 --> 00:11:57.340
And the whole world of Go players, like, you

00:11:57.340 --> 00:12:00.200
know, like all these people just missed like

00:12:00.200 --> 00:12:03.190
these like Choices that that AI found I think

00:12:03.190 --> 00:12:06.850
it's like just so cool and and so profound Yeah,

00:12:06.850 --> 00:12:08.129
I mean it a little bit happens in chess, but

00:12:08.129 --> 00:12:10.149
I think I think go is like more weird Yeah, I

00:12:10.149 --> 00:12:12.070
mean like I'm also kind of into chess. I'm less

00:12:12.250 --> 00:12:14.610
proficient at it and know it less well. But it's

00:12:14.610 --> 00:12:16.789
pretty fun watching, you know, the new chess

00:12:16.789 --> 00:12:18.950
programs that are trained off of again from scratch.

00:12:18.990 --> 00:12:20.929
And there's so much more aggressive than like

00:12:20.929 --> 00:12:23.389
the deep blues and stuff. And that makes me feel

00:12:23.389 --> 00:12:27.889
better about my unsound chess play. I can't speak

00:12:27.889 --> 00:12:31.809
to my go, but chess for sure. I used to I mean,

00:12:31.809 --> 00:12:33.730
I used to love playing chess. I wish I had more

00:12:33.730 --> 00:12:35.950
time. And then I also think about like poker

00:12:35.950 --> 00:12:38.450
and things like that and how much there is. Yes,

00:12:38.450 --> 00:12:42.240
there is like the mathematical statistical statistically

00:12:42.240 --> 00:12:44.759
correct, you know, move to make in every position.

00:12:45.179 --> 00:12:47.620
But there's also the game that you're playing,

00:12:47.620 --> 00:12:50.259
and there's the styles that you can have. So

00:12:50.259 --> 00:12:52.899
it's interesting to think about that. And the

00:12:52.899 --> 00:12:55.679
parallel that I'm thinking with, you know, frontier

00:12:55.679 --> 00:12:57.799
models and LLMs and things like that is like

00:12:58.799 --> 00:13:01.559
personas. And I'm thinking about that also almost

00:13:01.559 --> 00:13:03.980
as like the style in which you want something

00:13:03.980 --> 00:13:08.179
to be, you know, behaving in whatever environment

00:13:08.179 --> 00:13:10.559
you're in with them. So it's interesting, dependent

00:13:10.559 --> 00:13:12.659
on the context and dependent on like, you know,

00:13:12.759 --> 00:13:15.120
what the what the prompt is, you could have two

00:13:15.120 --> 00:13:18.320
very intelligent models, and they could respond,

00:13:18.600 --> 00:13:21.539
you know, they could respond differently. Yeah,

00:13:21.679 --> 00:13:23.259
well, but just making a different I mean, like,

00:13:23.320 --> 00:13:26.519
so go, there's one best style. Oh, it is. Okay.

00:13:26.720 --> 00:13:29.090
Oh, yeah. So like, you know, Because it's a full

00:13:29.090 --> 00:13:32.210
information game like chess, right? So like there's

00:13:32.210 --> 00:13:35.429
a best way This is best move at any point obviously

00:13:35.429 --> 00:13:38.669
and so I think you know, it's just I think that

00:13:38.669 --> 00:13:41.129
The models trained without human input now find

00:13:41.129 --> 00:13:44.389
it, you know find a better More effective style

00:13:44.389 --> 00:13:46.649
than the ones trained on on humans to sort of

00:13:46.649 --> 00:13:49.129
reveal that you know humans do some suboptimal

00:13:49.129 --> 00:13:52.129
Oh, I see what you're saying. I got you. I got

00:13:52.129 --> 00:13:53.970
you I guess that the thing that I was thinking

00:13:53.970 --> 00:13:57.100
about with chess was you could have the best

00:13:57.100 --> 00:14:00.759
move or in poker, you could have the best move

00:14:00.759 --> 00:14:03.080
given that you think somebody is going to then

00:14:03.080 --> 00:14:05.360
respond to you thinking that they're going to

00:14:05.360 --> 00:14:07.080
respond properly. But you never you don't know

00:14:07.080 --> 00:14:09.200
exactly what they're going to respond. So in

00:14:09.200 --> 00:14:11.299
chess, you could have a move where you're planning

00:14:11.299 --> 00:14:13.840
one, two, three moves ahead. But, you know, some

00:14:13.840 --> 00:14:16.360
of the best players are thinking multiple 10,

00:14:16.379 --> 00:14:19.480
15, you know, 10, 15 moves ahead. Is there is

00:14:19.480 --> 00:14:22.019
it like that with go also? Maybe. I don't know.

00:14:22.080 --> 00:14:24.019
I guess I don't like to. Yeah, it's okay. To

00:14:24.019 --> 00:14:30.720
play that way. I think generally relying on your

00:14:30.720 --> 00:14:33.259
opponent to make a mistake is a dangerous mindset.

00:14:33.480 --> 00:14:35.220
So what I'm talking about is the programs that

00:14:35.220 --> 00:14:37.500
try to play optimally. I think chess programs

00:14:37.500 --> 00:14:40.000
sometimes they try to train it to play like a

00:14:40.000 --> 00:14:42.500
human and make mistakes like a human. And I'm

00:14:42.500 --> 00:14:45.019
sure you could take advantage of human tendencies

00:14:45.019 --> 00:14:49.000
more effectively and crush them even more thoroughly.

00:14:49.740 --> 00:14:53.649
But I think these programs beat humans. 100 %

00:14:53.649 --> 00:14:56.149
of the time, you know, and like by just playing

00:14:56.149 --> 00:14:59.970
optimally, too. So, yeah, it's pretty amazing

00:14:59.970 --> 00:15:04.850
how the how how fast that has happened. I mean,

00:15:04.909 --> 00:15:07.870
like, I guess, you know, decades ago, people,

00:15:07.870 --> 00:15:10.309
I guess there were a lot of different feelings

00:15:10.309 --> 00:15:14.070
on the matter, but these sorts of games. Yeah,

00:15:14.289 --> 00:15:16.309
the the machine, the machine, the machine is

00:15:16.309 --> 00:15:19.269
going to the machine can win in all of these

00:15:19.269 --> 00:15:26.779
now. So was it the idea of the magic behind gradient

00:15:26.779 --> 00:15:29.419
descent, the ability for machines to get better

00:15:29.419 --> 00:15:31.919
over time, that was the thing that kind of attracted

00:15:31.919 --> 00:15:34.960
you to it? Or was it more than that? Well, in

00:15:34.960 --> 00:15:36.179
a way, that's all there is, right? But I mean,

00:15:36.259 --> 00:15:38.000
there's like what you draw from that, which is

00:15:38.000 --> 00:15:40.659
like, if computers can program other computers,

00:15:41.419 --> 00:15:44.559
then you can solve literally every problem that

00:15:44.559 --> 00:15:46.860
humans face, right? I just think when you have

00:15:46.860 --> 00:15:50.659
that recursive ability, Yeah, to automatically

00:15:50.659 --> 00:15:54.100
prove algorithms. I think that's just the most

00:15:54.100 --> 00:15:56.659
powerful technology that you could possibly build.

00:15:56.840 --> 00:15:59.120
It sort of subsumes every other endeavor that

00:15:59.120 --> 00:16:03.200
you might, you might try. Yeah, I kind of go

00:16:03.200 --> 00:16:07.840
for a hard pivot into fast forwarding to founding

00:16:07.840 --> 00:16:11.899
of weights and biases. What was the initial impetus

00:16:11.899 --> 00:16:15.360
there? Well, you know, I think it was I think

00:16:15.360 --> 00:16:18.740
in some ways for me personally, it was a reaction

00:16:18.740 --> 00:16:24.860
to what I liked and didn't like around CrowdFlower

00:16:24.860 --> 00:16:27.240
Figure 8. So I think I felt like CrowdFlower

00:16:27.240 --> 00:16:29.080
Figure 8, it's a good business. I mean, Scale

00:16:29.080 --> 00:16:33.340
AI has taken it and run really far with it. It's

00:16:33.340 --> 00:16:35.960
a good business, but it might not be the perfect

00:16:35.960 --> 00:16:38.960
business for me to be... You know, running like,

00:16:39.059 --> 00:16:40.659
I think I'm a little more oriented towards wanting

00:16:40.659 --> 00:16:43.240
to sell to developers, like wanting to make like

00:16:43.240 --> 00:16:46.220
high quality products. And, and I think I also

00:16:46.220 --> 00:16:47.840
just felt like a passion around it, like more

00:16:47.840 --> 00:16:49.820
than like, this seemed like a good business models.

00:16:49.960 --> 00:16:52.120
Like, you know, I really respect the people that

00:16:52.120 --> 00:16:54.899
are building these AI models. I'd like to help

00:16:54.899 --> 00:16:56.740
them. Like, I think like, you know, what I have

00:16:56.740 --> 00:16:58.820
to offer is like kind of understanding like how

00:16:58.820 --> 00:17:01.789
they work and like how to make their, um. their

00:17:01.789 --> 00:17:04.089
work work better. You know, my, my co -founder

00:17:04.089 --> 00:17:05.930
Sean and Chris and I, we were just really passionate

00:17:05.930 --> 00:17:07.390
about like moving forward to the state of the

00:17:07.390 --> 00:17:09.369
art of AI. We felt like what we had to offer

00:17:09.369 --> 00:17:11.809
was good, um, developer tools. And we had been

00:17:11.809 --> 00:17:13.710
pretty close to the GitHub founders and kind

00:17:13.710 --> 00:17:16.109
of watch them, you know, be successful. We also,

00:17:16.250 --> 00:17:17.470
I think one thing that was helpful to see was

00:17:17.470 --> 00:17:18.869
the GitHub founders at first, everyone's like,

00:17:18.869 --> 00:17:21.490
oh, developer tools. That's a small market. That's

00:17:21.490 --> 00:17:23.069
a stupid thing to work on, but it's sort of like

00:17:23.069 --> 00:17:24.869
developers became really powerful within the

00:17:24.869 --> 00:17:26.150
organizations. Cause like what they're doing

00:17:26.150 --> 00:17:28.670
is automating companies. I think what we were

00:17:28.670 --> 00:17:32.289
thinking. with Weights and Biases is these AI

00:17:32.289 --> 00:17:34.470
developers, even smaller market, it's like a

00:17:34.470 --> 00:17:37.329
subset of developers, but they're even more powerful

00:17:37.329 --> 00:17:39.630
within their organizations because they're automating

00:17:39.630 --> 00:17:42.769
the automation. So let's get on their side. Let's

00:17:42.769 --> 00:17:46.630
make stuff that's good for them. And that's going

00:17:46.630 --> 00:17:48.809
to be a great thing to offer the world. Yeah,

00:17:48.829 --> 00:17:52.309
definitely. So Weights and Biases was founded

00:17:52.309 --> 00:17:55.980
back in 2018. So it was a different... machine

00:17:55.980 --> 00:17:58.779
learning landscape then. I fell in love with

00:17:58.779 --> 00:18:04.720
Weights and Biases back in 2019. I used it to

00:18:04.720 --> 00:18:07.500
track experiments. I think I'm not sure if it

00:18:07.500 --> 00:18:09.859
was even earlier, but definitely one of my first

00:18:09.859 --> 00:18:12.119
one of my first big projects. I was like, I need

00:18:12.119 --> 00:18:15.059
to figure out a better way than writing all of

00:18:15.059 --> 00:18:17.700
my results down. And I want this to be automated.

00:18:17.920 --> 00:18:20.640
I was like, oh, OK, I could just add this these

00:18:20.640 --> 00:18:24.420
three lines of code. That sounds great. And now

00:18:24.420 --> 00:18:27.400
what it has evolved into. Oh, my God. It's like.

00:18:27.839 --> 00:18:31.599
So at the time, I was just, you know. logging

00:18:31.599 --> 00:18:33.619
classification reports and confusion matrices.

00:18:34.160 --> 00:18:36.000
Now I don't even know if it's like a standard

00:18:36.000 --> 00:18:38.740
thing. I put like my visualizations. I have like

00:18:38.740 --> 00:18:41.099
these data map plots that I put into weights

00:18:41.099 --> 00:18:45.859
and biases. Yeah, I have a lot of fun with it.

00:18:45.920 --> 00:18:48.660
I've told every machine learning person that

00:18:48.660 --> 00:18:51.839
I know how much I how much I really weights and

00:18:51.839 --> 00:18:54.579
biases has made my career so much better, like

00:18:54.579 --> 00:18:57.200
just so much better. So thank you. Thank you.

00:18:57.200 --> 00:18:59.779
Thanks for that. That kind of feels really good.

00:18:59.980 --> 00:19:03.420
Yeah, definitely. I hope you hear that because

00:19:03.420 --> 00:19:07.720
it's an amazing tool. Yeah, I guess maybe what

00:19:07.720 --> 00:19:09.680
happens is people start taking it for granted,

00:19:09.759 --> 00:19:12.680
but no, it has become ingrained in the way that

00:19:12.680 --> 00:19:18.250
we do work, 100%. So yeah. But what I was going

00:19:18.250 --> 00:19:20.910
to talk about or ask you is basically like, yeah,

00:19:20.930 --> 00:19:23.369
the machine learning landscape has changed. 2018,

00:19:23.690 --> 00:19:25.750
I think it was, I don't know what I was working

00:19:25.750 --> 00:19:28.549
on a lot of just like classification stuff. And

00:19:28.549 --> 00:19:33.890
now 2025, we have the LLM for the last like,

00:19:33.890 --> 00:19:36.049
what, three years, I guess. It's a very different

00:19:36.049 --> 00:19:38.529
landscape, the generative landscape. So how has

00:19:38.529 --> 00:19:42.170
the vision evolved for Weights and Biases? Can

00:19:42.170 --> 00:19:44.730
you talk like, yeah, just with the landscape

00:19:44.730 --> 00:19:47.839
changing and... Yeah, just you touch upon that.

00:19:48.500 --> 00:19:50.279
Yeah, I mean, the you know, the biggest change

00:19:50.279 --> 00:19:53.519
by far to the landscape, I think, is that, you

00:19:53.519 --> 00:19:55.900
know, when we started like. you'd mostly build

00:19:55.900 --> 00:19:58.180
your own model and like fine tuning was like

00:19:58.180 --> 00:19:59.839
some people would do it seemed like maybe a good

00:19:59.839 --> 00:20:01.660
idea but it was actually kind of unusual and

00:20:01.660 --> 00:20:04.180
now i think many many tasks you could just you

00:20:04.180 --> 00:20:06.279
know use an off -the -shelf lm you could go to

00:20:06.279 --> 00:20:07.819
open like if you want to translate something

00:20:07.819 --> 00:20:09.180
you don't need to build a translation model you

00:20:09.180 --> 00:20:11.460
can ask um opening literally just ask it to translate

00:20:11.460 --> 00:20:13.980
something right um and so so many of the the

00:20:13.980 --> 00:20:16.440
tasks now are just asking lms the right way and

00:20:16.440 --> 00:20:17.720
then we have like these new even more exciting

00:20:17.720 --> 00:20:19.720
tasks where it's like chaining the stuff together

00:20:19.720 --> 00:20:21.980
using tools which you call like agentic systems

00:20:21.980 --> 00:20:24.690
i think that's where um you know, most of the

00:20:24.690 --> 00:20:27.950
interesting innovation is I do think deep seek

00:20:27.950 --> 00:20:30.490
in particular cause the resurgence of people

00:20:30.490 --> 00:20:32.650
wanting to do their own reinforcement learning.

00:20:32.690 --> 00:20:36.390
So we are, you know, seeing actually more users

00:20:36.390 --> 00:20:38.750
coming on board to like weights and biases, original

00:20:38.750 --> 00:20:41.910
product and than ever, which is like really satisfying.

00:20:43.269 --> 00:20:45.369
But, you know, it's like a different style of

00:20:45.369 --> 00:20:46.849
thing, right? Your models are enormous. You're

00:20:46.849 --> 00:20:50.430
typically, you're always running on like GPUs.

00:20:51.369 --> 00:20:53.869
Often, you know a lot of GPUs running in a distributed

00:20:53.869 --> 00:20:56.069
way like that's like kind of table stakes like

00:20:56.069 --> 00:20:58.150
it was kind of exotic when we started like opening

00:20:58.150 --> 00:20:59.650
I was the only one doing distributed training

00:20:59.650 --> 00:21:03.509
for a long time on weights and biases and and

00:21:03.509 --> 00:21:06.450
now I think you know any Anybody really trying

00:21:06.450 --> 00:21:09.309
to do like meaningful scale is doing any distributed

00:21:09.309 --> 00:21:12.089
training of some some variety So so yeah, so

00:21:12.089 --> 00:21:14.170
look at what happened was we have this one kind

00:21:14.170 --> 00:21:16.259
of product that does experiment tracking that

00:21:16.259 --> 00:21:18.880
was really popular that we now call our models

00:21:18.880 --> 00:21:21.380
product, just to give it a name. That's a WB

00:21:21.380 --> 00:21:26.140
Python package you download. But we added a product

00:21:26.140 --> 00:21:29.960
called Weave that's designed just for running

00:21:29.960 --> 00:21:32.140
LLMs and kind of all the work that happens around

00:21:32.140 --> 00:21:34.140
that. And the two are actually great together,

00:21:34.180 --> 00:21:35.339
right? Because a lot of people, you want to build

00:21:35.339 --> 00:21:37.339
a model, then you want to run it. But I think

00:21:37.339 --> 00:21:39.599
that the number of people who would have a need

00:21:39.599 --> 00:21:42.309
for a product like Weave that could track an

00:21:42.309 --> 00:21:44.549
LLM that they got off the shelf is much bigger

00:21:44.549 --> 00:21:47.190
than the number of people building AI models.

00:21:47.309 --> 00:21:49.710
So I think what we actually saw was this democratization

00:21:49.710 --> 00:21:51.529
of AI that everybody had talked about and where

00:21:51.529 --> 00:21:55.930
anyone can really harness the power of AI. And

00:21:55.930 --> 00:21:57.990
then, you know, they get all the kind of drawbacks

00:21:57.990 --> 00:21:59.970
of AIT, right? Like, you know, just because you're

00:21:59.970 --> 00:22:01.829
running a third party LM, it's like still non

00:22:01.829 --> 00:22:04.170
-deterministic. You know, it's still like really

00:22:04.170 --> 00:22:05.970
hard, you know, like to know how to test it.

00:22:06.329 --> 00:22:07.230
Like, you know, you're never going to have a

00:22:07.230 --> 00:22:09.069
hundred percent test coverage and you have a

00:22:09.069 --> 00:22:10.930
non -deterministic, you know, LM underneath,

00:22:10.930 --> 00:22:12.710
you know, your system. And you're going to have

00:22:12.710 --> 00:22:14.289
to think a lot about like, okay, like what are

00:22:14.289 --> 00:22:16.089
the metrics? I'm actually trying to optimize

00:22:16.089 --> 00:22:18.170
here. How do I make them better? You know, what's

00:22:18.170 --> 00:22:19.910
different is like, you know, people kind of coming

00:22:19.910 --> 00:22:23.750
to that new without any machine learning training.

00:22:23.920 --> 00:22:25.400
You know, they're, they're not as comfortable

00:22:25.400 --> 00:22:27.500
with statistics as the average, you know, like

00:22:27.500 --> 00:22:30.960
ML person, and they don't even necessarily in

00:22:30.960 --> 00:22:33.119
the beginning, see the value and evaluations,

00:22:33.259 --> 00:22:34.259
right? Cause it's kind of annoying to set up

00:22:34.259 --> 00:22:37.000
evaluations. And you just wouldn't think of it.

00:22:37.039 --> 00:22:38.200
You'd be like, ah, just I'll see if it's good.

00:22:38.200 --> 00:22:40.240
And then I'll like, you know, ship it. And so

00:22:40.240 --> 00:22:41.539
I think in the early days of waste and biases,

00:22:42.000 --> 00:22:44.220
everyone, no one would be like, I shouldn't do

00:22:44.220 --> 00:22:46.460
evaluations. And we're like, how do we do. Right.

00:22:46.759 --> 00:22:48.740
I think with LMS, you kind of actually have to

00:22:48.740 --> 00:22:50.759
convince people to do evaluations. I do think

00:22:50.759 --> 00:22:53.049
there's a flip side of that though, like. I think

00:22:53.049 --> 00:22:55.369
ML researchers had this tendency to just look

00:22:55.369 --> 00:22:57.509
at aggregate statistics. And a lot of what our

00:22:57.509 --> 00:23:00.450
product would try to do is get you to look at

00:23:00.450 --> 00:23:02.410
individual examples and see what's happening,

00:23:02.430 --> 00:23:04.569
because there's a lot there usually. And usually,

00:23:04.589 --> 00:23:06.769
ML researchers don't do enough of that. I think

00:23:06.769 --> 00:23:09.170
you see the flip side of that in this LLM world,

00:23:09.890 --> 00:23:12.329
where people talk about testing by vibes. But

00:23:12.329 --> 00:23:14.329
testing by vibes is actually trying the thing,

00:23:14.369 --> 00:23:16.430
looking at some examples. It's a really good

00:23:16.430 --> 00:23:20.690
thing to do. So now that happens enough, maybe

00:23:20.690 --> 00:23:22.970
too much. And there's not enough to look at the

00:23:22.970 --> 00:23:24.809
aggregate statistics because, you know, it's

00:23:24.809 --> 00:23:27.630
like, Vibes will get you really far. Vibes is

00:23:27.630 --> 00:23:29.490
really good at preventing you from doing crazy

00:23:29.490 --> 00:23:32.210
things and having like these horrible, you know,

00:23:32.329 --> 00:23:33.710
like deployments and stuff. You know, if you're

00:23:33.710 --> 00:23:35.789
trying to like nudge inaccuracy higher, which

00:23:35.789 --> 00:23:38.049
is kind of where every project ends up and you

00:23:38.049 --> 00:23:40.490
want to get like from 69 % to 71 % accuracy to

00:23:40.490 --> 00:23:43.009
73 % to whatever, that's where you actually really

00:23:43.009 --> 00:23:44.630
need evaluations because you can't kind of make

00:23:44.630 --> 00:23:47.089
incremental steady progress without, you know,

00:23:47.250 --> 00:23:51.539
clear metrics that you're improving. Yeah, 100%.

00:23:51.539 --> 00:23:54.240
So I guess a couple of things. Yeah, so I loved

00:23:54.490 --> 00:23:56.730
When you guys came out with tables, I thought

00:23:56.730 --> 00:23:59.470
that that was really cool, because I was able

00:23:59.470 --> 00:24:02.869
to in weights and biases actually look and group

00:24:02.869 --> 00:24:06.150
my data and find and see particular examples.

00:24:06.549 --> 00:24:10.230
And I built and I've used weave. You know, we've

00:24:10.230 --> 00:24:12.470
yeah, we've is so cool. Some of the things that

00:24:12.470 --> 00:24:15.470
I really like about it is the tracing. So when

00:24:15.470 --> 00:24:18.589
you have, say, like multiple LLM calls, you have,

00:24:18.589 --> 00:24:20.349
you know, certain tracing. And what I really

00:24:20.349 --> 00:24:23.009
like is that you can see where the what parts

00:24:23.009 --> 00:24:25.960
of the code were touched with that area. And

00:24:25.960 --> 00:24:29.099
I haven't really been able to get that sort of

00:24:29.099 --> 00:24:32.299
detail anywhere else. So that's that's one of

00:24:32.299 --> 00:24:34.920
the reasons why I like I like using weave. And

00:24:34.920 --> 00:24:38.180
then, yeah, just to get into like evaluations,

00:24:38.339 --> 00:24:43.460
I think people the hard thing is it's just it's

00:24:43.460 --> 00:24:46.160
very hard to evaluate generative outputs. What

00:24:46.160 --> 00:24:49.339
makes a summary, one summary better than another

00:24:49.339 --> 00:24:51.420
summary, and you have to sort of think about

00:24:51.420 --> 00:24:53.500
different criteria of what it's going to be on,

00:24:53.500 --> 00:24:56.759
and then to go into your point. Yeah, like you

00:24:56.759 --> 00:24:58.980
have to have a sufficient, you have to have a

00:24:58.980 --> 00:25:01.480
robust evaluation system, because anytime you're

00:25:01.480 --> 00:25:03.839
making any kinds of changes into the system,

00:25:04.000 --> 00:25:06.119
whether it's prompt engineering, or whether it's

00:25:06.119 --> 00:25:08.119
using a new model, whatever changing hyper parameter,

00:25:08.180 --> 00:25:09.779
whatever you're doing, there are always trade

00:25:09.779 --> 00:25:12.500
offs that happen. And if you don't have a robust

00:25:12.500 --> 00:25:15.559
way of evaluating it going from something that

00:25:15.559 --> 00:25:17.859
looks good. You know, like, oh, I think it's

00:25:17.859 --> 00:25:20.200
good. I tried it out on 10 prompts. I tried it

00:25:20.200 --> 00:25:22.519
out on 10 examples. It looks good compared to

00:25:22.519 --> 00:25:24.960
like, oh no, like I need to get this performance

00:25:24.960 --> 00:25:28.880
up from 75 to 85. Any change that I'm making,

00:25:28.880 --> 00:25:31.220
I have to make sure. So what I've been doing

00:25:31.220 --> 00:25:34.759
is I just keep building my evaluation set, right?

00:25:34.759 --> 00:25:37.440
Just I get it done end to end and then I'll test

00:25:37.440 --> 00:25:39.619
10 things and then I'll make sure it still performs

00:25:39.619 --> 00:25:41.720
on those 10 things and it'll be 20, 30, 40. And

00:25:41.720 --> 00:25:43.259
then like all of a sudden you're like, whoa,

00:25:43.480 --> 00:25:45.740
I have like 150 things that I'm testing on this

00:25:45.740 --> 00:25:48.960
thing. saying, all right, it's still not gonna

00:25:48.960 --> 00:25:52.500
capture everything when I bring it to an actual

00:25:52.500 --> 00:25:54.940
user uses it because you can't ever, you have

00:25:54.940 --> 00:25:57.259
to continue to just kind of figure out how your

00:25:57.259 --> 00:26:00.500
users are using your model. But at least I understand

00:26:00.500 --> 00:26:02.960
there's no regressions in some of the capabilities

00:26:02.960 --> 00:26:05.980
that I wanted it to have from the get -go. So

00:26:05.980 --> 00:26:10.039
I think LLM observability is really hard. It's

00:26:10.039 --> 00:26:14.180
a challenging problem and I... It's hard because

00:26:14.180 --> 00:26:17.119
I know what it's like from your perspective,

00:26:17.579 --> 00:26:20.519
what Weights and Biases does is a general tool

00:26:20.519 --> 00:26:24.500
that can be applied to specific problems, which

00:26:24.500 --> 00:26:27.059
I guess is a lot of software obviously, but particularly

00:26:27.059 --> 00:26:31.230
for something where there's just... no real rule

00:26:31.230 --> 00:26:33.750
book, there's no like, oh, like check the F1

00:26:33.750 --> 00:26:35.369
score, you know, for like traditional machine

00:26:35.369 --> 00:26:38.789
learning. It's not that simple. So it's hard

00:26:38.789 --> 00:26:42.210
to know what to observe when it's unclear exactly

00:26:42.210 --> 00:26:44.250
like what people want to be building with it.

00:26:44.250 --> 00:26:46.630
So it must be a very, that must be very challenging.

00:26:46.789 --> 00:26:48.450
I would say I'd say I don't know, how do you

00:26:48.450 --> 00:26:51.450
deal with that? Like, where the goals of the

00:26:51.450 --> 00:26:54.250
people using your product are evolving so fast?

00:26:54.710 --> 00:26:56.890
Is that something that you think about? Totally.

00:26:56.930 --> 00:27:00.640
Yeah. I mean, I think it's I mean, it's exciting.

00:27:01.519 --> 00:27:03.400
It's like, you know, I think the, the entrepreneurial

00:27:03.400 --> 00:27:05.640
gets like really excited. You know, I think it's,

00:27:05.640 --> 00:27:08.160
you know, it, it kind of favors smaller companies

00:27:08.160 --> 00:27:09.900
in a way, right? Like when you need to like really

00:27:09.900 --> 00:27:12.980
iterate quickly. So I think like one of the funny

00:27:12.980 --> 00:27:15.200
challenges that I find myself in is trying to,

00:27:15.200 --> 00:27:18.259
you know, make like, um, you know, 250 plus person

00:27:18.259 --> 00:27:20.539
company, you know, recently acquired by a thousand

00:27:20.539 --> 00:27:23.059
person, you know, company, like just keep like,

00:27:23.059 --> 00:27:26.380
you know, thank you. Thank you. Um, yeah. You

00:27:26.380 --> 00:27:28.079
know, like I think we have to move really fast.

00:27:28.600 --> 00:27:30.700
And we have to like listen really carefully to

00:27:30.700 --> 00:27:32.720
what people are doing. But, you know, I think

00:27:32.720 --> 00:27:34.160
that you don't also, also you don't have to solve

00:27:34.160 --> 00:27:36.319
every problem, right? So like, you know, I think

00:27:36.319 --> 00:27:37.779
we talk to our customers all the time and we

00:27:37.779 --> 00:27:39.380
like look at their workflows and you see like,

00:27:39.559 --> 00:27:40.980
okay, people want to make these like small sets.

00:27:41.140 --> 00:27:43.640
They can quickly look at, you know, like LM is

00:27:43.640 --> 00:27:46.220
a judge is like a really popular strategy. There's

00:27:46.220 --> 00:27:48.240
different kind of patterns around that. There's

00:27:48.240 --> 00:27:50.440
also like, you know, making sure that like users

00:27:50.440 --> 00:27:52.960
can actually like, you know, put popular, the

00:27:52.960 --> 00:27:56.039
feedback back into weave. And there's often like.

00:27:56.430 --> 00:27:58.049
you know, sets of data where it's just like,

00:27:58.130 --> 00:27:59.730
you can never like get this one wrong. Like we

00:27:59.730 --> 00:28:01.809
never want like anything, you know, bad about

00:28:01.809 --> 00:28:04.029
our brand or like, you know, we don't anything

00:28:04.029 --> 00:28:06.230
like sexual in our, in our result, you know,

00:28:06.529 --> 00:28:08.369
like, so you get, you get like what ends up happening,

00:28:08.390 --> 00:28:10.210
I think in the same way that like, you know,

00:28:10.210 --> 00:28:12.049
we'd kind of know that we were dealing with,

00:28:12.049 --> 00:28:15.170
I think a serious like ML company when it'd have

00:28:15.170 --> 00:28:17.410
like thousands of metrics, right? Cause he's

00:28:17.410 --> 00:28:18.849
sort of like over time, you just get more and

00:28:18.849 --> 00:28:20.390
more metrics that you're like tracking around

00:28:20.390 --> 00:28:22.109
whatever you're doing. Cause it's like, in some

00:28:22.109 --> 00:28:24.400
sense, you're always optimizing one number. But

00:28:24.400 --> 00:28:25.880
nobody really thinks like that. You know, like

00:28:25.880 --> 00:28:27.740
they, they want to see all the different things

00:28:27.740 --> 00:28:29.059
that are happening. Cause they don't really know

00:28:29.059 --> 00:28:30.759
that what they're exactly optimizing. So they

00:28:30.759 --> 00:28:33.019
want to look like the same way as like with a

00:28:33.019 --> 00:28:34.880
balance where it's like, you know, over time,

00:28:34.960 --> 00:28:36.299
you know, there's all these different considerations

00:28:36.299 --> 00:28:38.839
that you have to live. What means good. And so

00:28:38.839 --> 00:28:41.420
you start adding more and more, um, you know,

00:28:41.480 --> 00:28:44.359
evaluation sets. And so I think that's, that's,

00:28:44.480 --> 00:28:45.819
that's where things move in. Like, you know,

00:28:45.900 --> 00:28:48.619
we don't have to. do everything for you. We want

00:28:48.619 --> 00:28:50.640
to do the most helpful stuff, but I think a lot

00:28:50.640 --> 00:28:51.900
of people should be writing their own evals.

00:28:52.000 --> 00:28:54.440
There's a lot of great third party open source

00:28:54.440 --> 00:28:57.720
evals libraries that we try to support really

00:28:57.720 --> 00:29:00.079
well. And then we also try to make it work easily

00:29:00.079 --> 00:29:02.039
out of the box so people can at least get a taste

00:29:02.039 --> 00:29:06.799
of the power of what we have. Very cool. We'll

00:29:06.799 --> 00:29:09.200
go into this one. What are you most excited about

00:29:09.200 --> 00:29:12.339
the future for Weights and Biases? Well, I think

00:29:12.339 --> 00:29:16.640
we're now inside of a public... uh, company with

00:29:16.640 --> 00:29:19.539
really big ambitions. And I think like, you know,

00:29:19.579 --> 00:29:21.380
one thing that's exciting is that, you know,

00:29:21.380 --> 00:29:23.640
they have, you know, they have a ton of resources

00:29:23.640 --> 00:29:25.940
available to build, you know, lots of stuff.

00:29:25.940 --> 00:29:28.339
So I think like, you know, we had to be really

00:29:28.339 --> 00:29:31.240
careful about like, where are we like aimed our,

00:29:31.240 --> 00:29:33.200
you know, resources, but we knew there's like

00:29:33.200 --> 00:29:36.160
tons of other like steps in the, in AI workflow

00:29:36.160 --> 00:29:38.339
that like, where we felt like, you know, we could

00:29:38.339 --> 00:29:40.519
do a better job of, of building great stuff,

00:29:40.559 --> 00:29:42.259
but you know, we also wanted to stay focused,

00:29:42.259 --> 00:29:43.819
but I think now it's like, there's a bigger scope

00:29:43.819 --> 00:29:47.109
here where kind of any any step in the AI workflow

00:29:47.109 --> 00:29:49.329
is kind of like fair game or something like we

00:29:49.329 --> 00:29:51.690
could build. That gets really exciting to be

00:29:51.690 --> 00:29:55.990
able to build more stuff. And I think we obviously

00:29:55.990 --> 00:29:59.410
don't want to only work with core. We've been

00:29:59.410 --> 00:30:01.049
really careful about always saying that we're

00:30:01.049 --> 00:30:02.910
going to work with every cloud. It's always going

00:30:02.910 --> 00:30:05.630
to work with all the different infrastructure

00:30:05.630 --> 00:30:08.509
providers out there. But it's kind of interesting

00:30:08.509 --> 00:30:12.690
to actually see real life infrastructure and

00:30:12.690 --> 00:30:15.579
hardware. Um, deeply cause it does show you,

00:30:15.700 --> 00:30:17.900
there's like more stuff you could surface. Um,

00:30:18.000 --> 00:30:19.339
that I think would like kind of help the end

00:30:19.339 --> 00:30:21.400
user, especially the AI researcher. Like I think

00:30:21.400 --> 00:30:23.680
we could do more to like, let them know like,

00:30:23.859 --> 00:30:25.640
Hey, you know, weird things are happening in

00:30:25.640 --> 00:30:27.680
your, in your hardware stack and your training

00:30:27.680 --> 00:30:30.180
that you may want to take a look at. Um, we have

00:30:30.180 --> 00:30:31.859
a conference coming up in a, in a couple of weeks

00:30:31.859 --> 00:30:34.240
called fully connected. Um, and we're going to

00:30:34.240 --> 00:30:35.799
announce a whole bunch of like integrations there

00:30:35.799 --> 00:30:37.460
that that's kind of the tip of the iceberg of

00:30:37.460 --> 00:30:40.440
what I'm, I'm excited about. Cool. Yeah, that's

00:30:40.440 --> 00:30:42.759
exciting. We met last year at Fully Connected.

00:30:42.759 --> 00:30:47.200
That's a great conference. Yeah. In terms of,

00:30:47.200 --> 00:30:51.460
you know, joining a larger company and, you know,

00:30:51.660 --> 00:30:54.400
understanding more of the integration and scaling

00:30:54.400 --> 00:30:57.289
and things like that. So, yeah, you're thinking

00:30:57.289 --> 00:31:01.269
there's just a more diverse problem set or more

00:31:01.269 --> 00:31:04.950
areas where you could kind of help your end users

00:31:04.950 --> 00:31:07.829
or your understanding things better. Yeah, like

00:31:07.829 --> 00:31:10.630
what excites you about that? Well, I think it's

00:31:10.630 --> 00:31:13.890
like, you know, we go head to head against like

00:31:13.890 --> 00:31:16.049
Amazon. Amazon, you know, we're friendly with

00:31:16.049 --> 00:31:18.529
Amazon, but they have a competing product for

00:31:18.529 --> 00:31:21.549
like most of the stuff. that we do, and mostly

00:31:21.549 --> 00:31:25.130
their product is garbage. I don't even know if

00:31:25.130 --> 00:31:27.569
they would like, if you privately meet these

00:31:27.569 --> 00:31:30.509
people, they'll tell you that they think the

00:31:30.509 --> 00:31:32.769
weights and biases product is better. Boy, I'm

00:31:32.769 --> 00:31:34.750
not making any friends with this comment, but

00:31:34.750 --> 00:31:37.710
honestly, I think we've picked off a few things

00:31:37.710 --> 00:31:39.970
that done them really well. I think at an executive

00:31:39.970 --> 00:31:42.250
level though, people want to buy a complete solution.

00:31:42.529 --> 00:31:45.529
No developer wants that, but for some reason...

00:31:45.480 --> 00:31:47.099
I shouldn't say for some reason. I understand

00:31:47.099 --> 00:31:49.180
why. Like executives and companies kind of want

00:31:49.180 --> 00:31:50.819
to buy like one complete solution, have it be

00:31:50.819 --> 00:31:54.819
like cohesive and sell it. And I think like,

00:31:54.839 --> 00:31:56.660
you know, it's kind of a danger for weights and

00:31:56.660 --> 00:31:59.319
biases, the product, right? Like, you know, people

00:31:59.319 --> 00:32:00.980
would use like MLflow because they're like a

00:32:00.980 --> 00:32:02.900
Databricks shop and they're just like, look,

00:32:02.980 --> 00:32:05.160
I wish I could use weights and biases, but like,

00:32:05.160 --> 00:32:07.119
you know, Databricks comes with like MLflow for

00:32:07.119 --> 00:32:08.900
free and my boss is saying I have to use it,

00:32:08.900 --> 00:32:11.140
you know? Like, I think there's a lot of power

00:32:11.140 --> 00:32:14.180
immediately. in bringing these two companies

00:32:14.180 --> 00:32:16.180
together, like CoreWeave and Weights and Biases,

00:32:16.400 --> 00:32:21.319
now we can offer more complete solutions. CoreWeave

00:32:21.319 --> 00:32:23.180
is also incredibly fast, like revenue growth,

00:32:23.359 --> 00:32:26.259
far faster than us because of this amazing demand

00:32:26.259 --> 00:32:28.559
for hardware and they execute really well. And

00:32:28.559 --> 00:32:32.240
so we can also use some of their resources to

00:32:32.240 --> 00:32:34.220
build a more complete solution, which is probably

00:32:34.220 --> 00:32:36.500
where the world is ultimately going. So I think

00:32:36.500 --> 00:32:40.490
we could have gone on Um, alone, but it's not

00:32:40.490 --> 00:32:41.750
free. You're always kind of like looking for

00:32:41.750 --> 00:32:44.289
ways to, to like get an edge because it's these

00:32:44.289 --> 00:32:46.769
markets tend to be like winner take all or winner

00:32:46.769 --> 00:32:49.150
take most. And so like, you know, I'm pretty

00:32:49.150 --> 00:32:50.809
focused on like, okay, how do we like completely

00:32:50.809 --> 00:32:53.230
dominate this market? Like not end up in some

00:32:53.230 --> 00:32:58.789
sort of niche, um, category. Yeah. Sort of transitioning

00:32:58.789 --> 00:33:01.730
into the entrepreneurship, uh, parts of things.

00:33:02.230 --> 00:33:06.759
It's interesting thinking about. tech solutions

00:33:06.759 --> 00:33:10.380
and how there are many solutions obviously for

00:33:10.380 --> 00:33:12.940
many problems that's like, you know, the exciting

00:33:12.940 --> 00:33:16.140
thing about being in this in any, you know, market,

00:33:16.140 --> 00:33:18.740
but Basically, like Weights and Biases has a

00:33:18.740 --> 00:33:21.240
lot of competitors. There's open source competitors.

00:33:21.700 --> 00:33:25.019
There's other third parties. There's complete

00:33:25.019 --> 00:33:26.839
companies around it. There are companies where

00:33:26.839 --> 00:33:29.400
they're just doing things. I mean, you know,

00:33:29.759 --> 00:33:32.299
you spoke about Amazon a little bit. It's like,

00:33:32.359 --> 00:33:34.319
yeah, like Amazon has their own foundation models,

00:33:34.660 --> 00:33:36.839
too. You know, but like, I don't know a single

00:33:36.839 --> 00:33:39.819
person that's using them. And they also invested

00:33:39.819 --> 00:33:42.640
in other foundation models. So they have these

00:33:42.640 --> 00:33:45.000
huge, huge incentives. But to go back to the

00:33:45.000 --> 00:33:47.539
other point, it's There's an interesting thing.

00:33:48.079 --> 00:33:49.660
If you think about all the problems that are

00:33:49.660 --> 00:33:52.400
flowing down, you try to get a bucket and you

00:33:52.400 --> 00:33:54.980
try to find that bucket that you're going to

00:33:54.980 --> 00:33:56.799
capture a lot of the problems that you're getting.

00:33:57.259 --> 00:33:59.019
But then at a certain point, if you're a company

00:33:59.019 --> 00:34:00.980
that's buying these vendors, it's like, well,

00:34:00.980 --> 00:34:04.460
how many buckets am I going to have? I want one

00:34:04.460 --> 00:34:06.339
solution that's going to solve all of these things.

00:34:06.640 --> 00:34:11.199
And then sometimes you might not be able to focus

00:34:11.199 --> 00:34:16.090
narrowly on the problem. I know you definitely

00:34:16.090 --> 00:34:17.869
think somewhat about this, but I'm curious, like,

00:34:17.889 --> 00:34:21.750
what's your take on this whole there being many

00:34:21.750 --> 00:34:24.110
solutions for things and like tech consolidation?

00:34:24.309 --> 00:34:26.690
Yeah, like you sort of touched upon it, but do

00:34:26.690 --> 00:34:29.670
you have you do you think about that in the industry

00:34:29.670 --> 00:34:31.710
that you're in? Yeah, totally. I mean, I think

00:34:31.710 --> 00:34:34.250
what Marcus tend to do is like when there's some

00:34:34.250 --> 00:34:37.329
like new problem and you need new capabilities,

00:34:37.349 --> 00:34:39.929
that tends to favor smaller. companies. So you

00:34:39.929 --> 00:34:43.130
get this kind of Cambrian explosion of different

00:34:43.130 --> 00:34:46.269
options and the sort of biggest ones tend to

00:34:46.269 --> 00:34:49.989
consolidate. And so I think we're kind of seeing

00:34:49.989 --> 00:34:52.429
consolidation of some stuff. I think there's

00:34:52.429 --> 00:34:54.750
a time when MLOps is all the rage. You don't

00:34:54.750 --> 00:34:57.429
hear about that anymore. I think that's definitely

00:34:57.429 --> 00:35:02.130
kind of consolidating market. But then there's

00:35:02.130 --> 00:35:05.679
also a lot of new like LM, you know, functionality

00:35:05.679 --> 00:35:07.360
around like, I don't know, like, like routing,

00:35:07.539 --> 00:35:08.800
like, you know, how many companies need like

00:35:08.800 --> 00:35:11.880
a router today? Like, probably not a lot, like,

00:35:12.000 --> 00:35:14.099
will they need a router in the future? Probably

00:35:14.099 --> 00:35:18.119
most will want one, right? So that's like a really

00:35:18.119 --> 00:35:21.099
interesting, you know, category of thing. And

00:35:21.099 --> 00:35:23.440
there's a bunch of companies there. And, you

00:35:23.440 --> 00:35:27.480
know, I think There'll probably be more. And

00:35:27.480 --> 00:35:29.119
then over time, there'll be less, right? Because

00:35:29.119 --> 00:35:31.139
maybe that's not really a standalone business.

00:35:31.239 --> 00:35:33.500
I'm not sure. Maybe the ones that are doing it

00:35:33.500 --> 00:35:36.260
will expand into other categories. I think that's

00:35:36.260 --> 00:35:40.280
just life and technology. But I think with AI

00:35:40.280 --> 00:35:41.619
and the speed of innovation, it just happens

00:35:41.619 --> 00:35:46.599
way faster than it used to. Yeah. I think I'm

00:35:46.599 --> 00:35:48.639
stealing a question that you asked your guests.

00:35:50.159 --> 00:35:53.440
What's a free idea? You asked that, right? What's

00:35:53.440 --> 00:35:56.059
an idea if you had any time in the world that

00:35:56.059 --> 00:35:58.699
you think... Yeah, it's funny. I feel like I

00:35:58.699 --> 00:36:04.099
ask it more in a non -commercial context where

00:36:04.099 --> 00:36:06.099
I'm actually just curious. Because I always think,

00:36:06.260 --> 00:36:08.300
if I'm not doing this, what would I really do?

00:36:08.440 --> 00:36:12.260
It's a good question to ask yourself what's interesting

00:36:12.260 --> 00:36:17.090
out there. In terms of startup ideas, Man, it's

00:36:17.090 --> 00:36:19.050
like, I don't know. It just seems like such a

00:36:19.050 --> 00:36:23.090
fun time to be an entrepreneur right now. Because

00:36:23.090 --> 00:36:26.210
I feel like, I mean, this is not a secret, but

00:36:26.210 --> 00:36:29.690
you take this into any application and it just

00:36:29.690 --> 00:36:32.170
still blows people's minds. People just do not

00:36:32.170 --> 00:36:34.469
connect the dots between LLM generally and LLM

00:36:34.469 --> 00:36:37.829
applied to a particular problem people have.

00:36:37.889 --> 00:36:40.599
And so I'm just seeing all this. You know success

00:36:40.599 --> 00:36:42.280
in that enthusiasm. I think people kind of think

00:36:42.280 --> 00:36:43.659
they're late Like my friends will talk to me

00:36:43.659 --> 00:36:45.739
be like, oh am I too late to this? No, man, you're

00:36:45.739 --> 00:36:47.539
like still early It's amazing, you know, just

00:36:47.539 --> 00:36:49.820
like right like I think lms applied to anything.

00:36:49.820 --> 00:36:52.539
It's probably probably like a good idea honestly,

00:36:52.539 --> 00:36:55.659
um, you know, I don't know like I think it's

00:36:55.659 --> 00:36:58.420
like good to pick ideas that you kind of care

00:36:58.420 --> 00:37:03.380
about um, I mean Again, i'm not telling anybody

00:37:03.380 --> 00:37:04.920
the things that they probably don't know but

00:37:04.920 --> 00:37:08.400
it's like um The progress in robotics right now

00:37:08.400 --> 00:37:13.039
is unbelievably exciting. That's obviously going

00:37:13.039 --> 00:37:15.760
to be this huge thing. Again, I think you might

00:37:15.760 --> 00:37:17.940
think you're late, but you're early if you're

00:37:17.940 --> 00:37:23.300
looking into that. That's what I'm intrigued

00:37:23.300 --> 00:37:27.500
by. When you're doing a company, you really don't

00:37:27.500 --> 00:37:34.090
have time for anything else. I know. The potential

00:37:34.090 --> 00:37:38.010
for robotics is just too exciting. I mean, being

00:37:38.010 --> 00:37:41.909
able to combine sort of all of the power of LLMs

00:37:41.909 --> 00:37:44.829
along with, you know, computer vision and all

00:37:44.829 --> 00:37:47.210
that, I think that's a really exciting, really,

00:37:47.210 --> 00:37:49.769
really exciting field. I'm so curious to see

00:37:49.769 --> 00:37:52.369
what's going to be coming out in the next couple

00:37:52.369 --> 00:37:55.090
years and what people are currently working on.

00:37:55.789 --> 00:37:58.210
Zooming out to just like kind of machine learning

00:37:58.210 --> 00:38:00.070
in general and you know, maybe we touched upon

00:38:00.070 --> 00:38:02.289
it Maybe we didn't but what do you think is an

00:38:02.289 --> 00:38:05.250
important question that you believe remains unanswered

00:38:05.250 --> 00:38:08.690
in machine learning? I mean, there's a lot I

00:38:08.690 --> 00:38:12.210
think like I Don't know what then I'm like kind

00:38:12.210 --> 00:38:16.269
of like intrigued by is like you know if you

00:38:16.269 --> 00:38:19.909
ran if you ran back like history like a thousand

00:38:19.909 --> 00:38:23.050
times, you know, like how much of, you know,

00:38:23.130 --> 00:38:26.389
what we do around like a transformer architecture

00:38:26.389 --> 00:38:28.510
would be consistent and how much is a sort of

00:38:28.510 --> 00:38:30.389
product of like, you know, how we, how we do

00:38:30.389 --> 00:38:31.550
it. Like I think like, you know, there's just

00:38:31.550 --> 00:38:33.469
not a lot of like appetite for experimenting

00:38:33.469 --> 00:38:34.909
with stuff and people don't tend to write it

00:38:34.909 --> 00:38:36.429
down when they get just the same result. Like

00:38:36.429 --> 00:38:39.739
I feel like. There used to be LSTMs, GRUs, all

00:38:39.739 --> 00:38:42.079
these different architectures. I even remember

00:38:42.079 --> 00:38:44.099
trying to program these things by hand and getting

00:38:44.099 --> 00:38:47.039
it a little bit wrong, and it's still fine. So

00:38:47.039 --> 00:38:51.300
I wonder a lot about what are the core bits of

00:38:51.300 --> 00:38:53.860
this architecture that really matter. And then,

00:38:53.980 --> 00:38:56.920
of course, we now design our chips around this

00:38:56.920 --> 00:38:58.559
transformer architecture, so it's still locked

00:38:58.559 --> 00:39:01.059
in. You wouldn't want to change the aspects of

00:39:01.059 --> 00:39:04.699
it. But I think that's maybe an intellectual...

00:39:04.730 --> 00:39:08.210
Question that I I wonder about a lot and I think

00:39:08.210 --> 00:39:11.769
and there's been an explosion of Research and

00:39:11.769 --> 00:39:13.690
reinforcement learning right now, which I think

00:39:13.690 --> 00:39:16.269
is totally makes sense, but I'm having trouble

00:39:16.269 --> 00:39:19.110
Comprehending I'm talking way like this is sort

00:39:19.110 --> 00:39:20.210
of trying to like understand right now. It's

00:39:20.210 --> 00:39:22.929
like what what about these different? Like reward

00:39:22.929 --> 00:39:26.429
functions make them effective or not. I Don't

00:39:26.429 --> 00:39:28.329
know. I'm confused. Maybe somebody understands

00:39:28.329 --> 00:39:31.150
it. But yeah, I'm more confused these days than

00:39:31.150 --> 00:39:36.489
I was in the past Yeah, we have a mutual friend

00:39:36.489 --> 00:39:39.349
that would understand. John Schulman would understand

00:39:39.349 --> 00:39:41.630
proximal policy optimization. But anyway, going

00:39:41.630 --> 00:39:44.769
back to the other point, Transformers, and I

00:39:44.769 --> 00:39:46.849
think about it too, I think about this with everything

00:39:46.849 --> 00:39:51.769
in life also, is it just a local maximum or is

00:39:51.769 --> 00:39:54.650
it a global, you know, or, you know, is it a

00:39:54.650 --> 00:39:58.210
local minimum or is it a global? Like, is this

00:39:58.210 --> 00:40:00.909
the most optimal? And then I just was speaking

00:40:00.909 --> 00:40:04.579
about it, but also like, yeah, like, The 2017

00:40:04.579 --> 00:40:08.199
architecture is not the 2025 architecture and

00:40:08.199 --> 00:40:10.760
that's because the bet was placed and the bet

00:40:10.760 --> 00:40:12.480
was placed and the infrastructure was put into

00:40:12.480 --> 00:40:14.800
place and the and the chips were made and the

00:40:14.800 --> 00:40:16.659
ecosystems were created and then you start iterating

00:40:16.659 --> 00:40:19.500
on something and if you make a choice and you

00:40:19.500 --> 00:40:23.079
iterated on on it for eight years and You start

00:40:23.079 --> 00:40:24.860
to solve all of those problems that people were

00:40:24.860 --> 00:40:26.400
having it's like you're gonna get it to a place

00:40:26.400 --> 00:40:29.829
where it's really good and then we are seeing

00:40:29.829 --> 00:40:31.329
that we're that we're probably going to hit a

00:40:31.329 --> 00:40:34.409
ceiling but I think that already it's unlocked

00:40:34.409 --> 00:40:37.590
so much it's unlocked so much and just like what

00:40:37.590 --> 00:40:40.190
going to your previous answer just applying LLMs

00:40:40.190 --> 00:40:42.150
anywhere is there's you're going to be you're

00:40:42.150 --> 00:40:43.309
going to be able to you're going to be able to

00:40:43.309 --> 00:40:45.429
get value so it's going to be very interesting

00:40:45.429 --> 00:40:48.150
to see sort of what what happens this is another

00:40:48.150 --> 00:40:51.090
one that I like to ask how do you view the gap

00:40:51.090 --> 00:40:56.409
between the hype and the reality of AI? Yeah,

00:40:56.409 --> 00:40:58.789
okay, so like hype is tricky because I live in

00:40:58.789 --> 00:41:04.530
San Francisco and I Don't really know like the

00:41:04.530 --> 00:41:07.429
level of hype Like out in the world like it seems

00:41:07.429 --> 00:41:09.829
like very like uneven like I'm always kind of

00:41:09.829 --> 00:41:11.889
surprised that like resonates with my friends

00:41:11.889 --> 00:41:15.989
You know another in other Geos and and not But

00:41:15.989 --> 00:41:18.170
look honestly if you zoom out even a little bit

00:41:18.170 --> 00:41:20.869
all this stuff is so underhyped Like you can't

00:41:20.869 --> 00:41:22.750
you can't hype it enough. I mean, it's just like

00:41:22.750 --> 00:41:25.230
it's just amazing, you know, like it's incredible

00:41:25.369 --> 00:41:27.050
This technology works really well. It affects

00:41:27.050 --> 00:41:29.869
so many industries. People are maybe disappointed

00:41:29.869 --> 00:41:32.489
that it didn't solve every single one of their

00:41:32.489 --> 00:41:35.650
business problems right now. And it's like, relax.

00:41:35.929 --> 00:41:41.530
Obviously, it will soon. Just wait. Just wait.

00:41:42.030 --> 00:41:45.920
Wait a month. Yeah, exactly. There'll be a new

00:41:45.920 --> 00:41:49.360
model next month. I can do it. I mean, one of

00:41:49.360 --> 00:41:51.280
the exciting things, I don't know if you've had

00:41:51.280 --> 00:41:53.380
a chance. Have you had a chance to play with

00:41:53.380 --> 00:41:57.539
any of the AI assisted coding like cursor or

00:41:57.539 --> 00:41:58.960
things like that? Yeah, of course. Come on. Yeah.

00:41:59.039 --> 00:42:00.539
If you're not doing that, what's wrong with you?

00:42:00.860 --> 00:42:02.920
I honestly like when I come just an engineer.

00:42:03.420 --> 00:42:07.360
Yeah. Yeah. Yeah. I mean, I think there's like,

00:42:07.380 --> 00:42:11.000
I think cursor code is like so much fun. Yeah,

00:42:11.000 --> 00:42:14.409
I think that. I think that really is a really

00:42:14.409 --> 00:42:17.429
well -done product. Sorry, I mean, Cloud Code.

00:42:18.110 --> 00:42:21.090
Oh, Cloud Code. Yeah, Cloud Code, I think it's

00:42:21.090 --> 00:42:24.909
really delightful. I think Cursor is also really

00:42:24.909 --> 00:42:27.889
well -done. I mean, Windsurf. I don't know, I

00:42:27.889 --> 00:42:31.489
like them all, honestly. I think they change

00:42:31.489 --> 00:42:36.610
so fast and they're just awesome. It's funny,

00:42:36.789 --> 00:42:40.210
I'm coming into... You know core weave and I'm

00:42:40.210 --> 00:42:42.210
and they like, you know that for them like kubernetes.

00:42:42.269 --> 00:42:45.050
They like know it so well, and I'm just like

00:42:45.050 --> 00:42:46.889
Like they're like, oh here's some machines, but

00:42:46.889 --> 00:42:48.670
it's just like a kubernetes cluster. I'm like,

00:42:48.690 --> 00:42:51.170
oh man Like I never learned how to do this, you

00:42:51.170 --> 00:42:54.469
know, and so Yeah, I started asking like cursor

00:42:54.469 --> 00:42:56.690
like hey just like to play my stuff like in this

00:42:56.690 --> 00:42:58.170
kubernetes cluster And it's just sort of like

00:42:58.170 --> 00:43:00.110
okay, you know, like has a couple questions and

00:43:00.110 --> 00:43:01.769
then it's just like firing off jobs I'm like,

00:43:01.769 --> 00:43:03.969
yeah, that's amazing. I love it. Yeah, and then

00:43:03.969 --> 00:43:06.130
meanwhile my daughter, you know, it's me is like

00:43:06.130 --> 00:43:08.920
Claude code together To just like vibe code like

00:43:08.920 --> 00:43:10.900
weird games, you know, and and I was like, you

00:43:10.900 --> 00:43:12.280
know, I tried with cloud code. I mean, this is

00:43:12.280 --> 00:43:14.199
like going to be so dated in a few months because

00:43:14.199 --> 00:43:16.440
everyone knows. But I was like, hey, like, can

00:43:16.440 --> 00:43:18.139
you like deploy this so my daughter can show

00:43:18.139 --> 00:43:20.559
it to her friends? You know, and it's just like,

00:43:20.559 --> 00:43:22.440
oh, yeah, make a Netlify account and I'll like

00:43:22.440 --> 00:43:25.440
put the app in there for you. And it really did.

00:43:25.500 --> 00:43:28.840
Like one, you know, like what's good is so cool.

00:43:29.019 --> 00:43:33.139
It's so cool. Yeah. Yeah. I find myself like.

00:43:33.280 --> 00:43:36.179
anyone in my company that wants to work on something

00:43:36.179 --> 00:43:38.619
or like has an idea, I'm just like, can you just

00:43:38.619 --> 00:43:40.380
download like, can you just download cursor?

00:43:40.380 --> 00:43:42.980
It's not that simple. You need to know how to

00:43:42.980 --> 00:43:45.019
code to be in a code editor like you do, like

00:43:45.019 --> 00:43:48.639
you still kind of do. But there are other tools

00:43:48.639 --> 00:43:49.900
that are out there. What is Lovable? I don't

00:43:49.900 --> 00:43:51.659
know, my dad uses Lovable. He was making these

00:43:51.659 --> 00:43:55.539
crazy games and showing them to me. And yeah,

00:43:55.619 --> 00:43:57.400
I mean, like, yeah, Lovable is great. I mean,

00:43:57.400 --> 00:43:58.679
there's other ones too, but that's the one that

00:43:58.679 --> 00:44:06.460
he was. Yeah. Replet has one zero V zero. There's

00:44:06.460 --> 00:44:09.079
so many and I'm sure in like, you know, a couple

00:44:09.079 --> 00:44:11.500
of weeks, there'll be another there'll be another

00:44:11.500 --> 00:44:14.099
dozen that'll be doing something that's mind

00:44:14.099 --> 00:44:17.199
boggling. It's it's it's amazing. I mean, the

00:44:17.199 --> 00:44:21.090
nature of work is changing. I just there are

00:44:21.090 --> 00:44:24.269
so many people that view that look at the negatives

00:44:24.269 --> 00:44:26.829
of it, but there are so many positives to it.

00:44:26.989 --> 00:44:29.190
Also, the democratization, like you were talking

00:44:29.190 --> 00:44:31.889
about earlier, many more people can do many more

00:44:31.889 --> 00:44:34.929
things. And it's just it's so it's so much. It's

00:44:34.929 --> 00:44:36.929
such an exciting time. It's it's such an exciting

00:44:36.929 --> 00:44:41.309
time to be in. Yeah. What what's your daughter

00:44:41.309 --> 00:44:45.090
deploy? No, it's in some links. I mean, we're

00:44:45.090 --> 00:44:47.269
we're just five. So the mind of a five year old

00:44:47.269 --> 00:44:51.239
is like a weird. weird place, you know, so yeah,

00:44:51.539 --> 00:44:55.579
but that's so cool that she was able to do it.

00:44:56.900 --> 00:45:02.360
Very cool. Where do you see things in three years

00:45:02.360 --> 00:45:06.719
or five years? What's your take on like software

00:45:06.719 --> 00:45:09.019
developers? Do you think that there's going to

00:45:09.019 --> 00:45:10.619
be like you think there's really going to be

00:45:10.619 --> 00:45:13.019
a real hit to software developers? Or do you

00:45:13.019 --> 00:45:14.500
think the nature of work is going to change?

00:45:14.539 --> 00:45:17.820
What's your take on there? Look, I don't have

00:45:17.820 --> 00:45:20.039
any special insight. I do think like one thing

00:45:20.039 --> 00:45:21.980
that I'm seeing that's kind of I've been thinking

00:45:21.980 --> 00:45:26.300
about a lot and this is it seems like You might

00:45:26.300 --> 00:45:27.860
have thought that this sort of be like more of

00:45:27.860 --> 00:45:31.679
a democratization of Code and certainly that's

00:45:31.679 --> 00:45:33.360
true, right? It seems like non developers can

00:45:33.360 --> 00:45:35.599
like make stuff But it seems like the more dominant

00:45:35.599 --> 00:45:38.000
effect is it's making the best developers even

00:45:38.000 --> 00:45:42.780
more productive Than not the best developers

00:45:42.780 --> 00:45:44.380
like I don't know when you like meet the teams

00:45:44.380 --> 00:45:46.159
at some of these companies like, you know cursor

00:45:46.159 --> 00:45:48.880
You know kodium or madness or something? It's

00:45:48.880 --> 00:45:51.119
just like it seems like a small number of like

00:45:51.119 --> 00:45:54.099
incredibly hard -working effective people and

00:45:54.099 --> 00:45:56.760
so that's like sort of seems like the winning

00:45:56.760 --> 00:45:58.360
Strategy like it sort of seems like everything

00:45:58.360 --> 00:46:01.820
we do kind of increases Inequality like unfortunately

00:46:01.820 --> 00:46:04.280
and that and that it seems like that's what's

00:46:04.280 --> 00:46:06.500
what's happening, you know right now and it's

00:46:06.500 --> 00:46:08.179
very hard to predict like I think like if you

00:46:08.179 --> 00:46:11.699
told me It would be able to automate a lot of

00:46:11.699 --> 00:46:14.340
code tasks. I think I would have thought, you

00:46:14.340 --> 00:46:15.579
know, the upshot of that would be like, wow,

00:46:15.619 --> 00:46:17.539
now I can like hire like more developers, be

00:46:17.539 --> 00:46:21.159
even, you know, more effective. But I'm not so

00:46:21.159 --> 00:46:23.619
sure that that's like the right thing to do right

00:46:23.619 --> 00:46:25.599
now. Like even with, you know, with what we're

00:46:25.599 --> 00:46:27.800
seeing, like it might be, you know, now like

00:46:27.800 --> 00:46:29.659
really small teams or even like one person can

00:46:29.659 --> 00:46:31.780
be more effective. I definitely think like. you

00:46:31.780 --> 00:46:33.599
know, engineers have like more of a product mindset

00:46:33.599 --> 00:46:35.320
can be more effective. Cause the way a prop,

00:46:35.320 --> 00:46:37.440
probably just kind of like a prompt engineer,

00:46:37.440 --> 00:46:42.119
you know, for fresh years, right? So, um, it

00:46:42.119 --> 00:46:46.119
seems like it seems like moving up the, the sort

00:46:46.119 --> 00:46:50.019
of stack is probably, you know, probably, probably

00:46:50.019 --> 00:46:55.079
like favors that. Um, but yeah, I mean, um, I

00:46:55.079 --> 00:46:56.659
don't know. I think about it with like my kids

00:46:56.659 --> 00:46:59.030
too. It's like, okay, like should they, Like

00:46:59.030 --> 00:47:00.869
learn to code and stuff like I'm kind of like

00:47:00.869 --> 00:47:04.250
okay like learn to code as we do today If it's

00:47:04.250 --> 00:47:06.949
like fun for you in the same way that I think

00:47:06.949 --> 00:47:08.730
like woodworking projects are like sometimes

00:47:08.730 --> 00:47:11.289
fun, right? Maybe we'll you know, we'll have

00:47:11.289 --> 00:47:15.809
like artisanal You know libraries like charming

00:47:15.809 --> 00:47:22.719
bugs Yeah That's funny. No, it's a couple of

00:47:22.719 --> 00:47:25.980
good points there. I mean, well, I'll start with

00:47:25.980 --> 00:47:28.940
the last one. It's like, you know, like I had

00:47:28.940 --> 00:47:31.739
to like build a clock at one point in my life.

00:47:31.980 --> 00:47:33.719
And no, I never need to know how to actually

00:47:33.719 --> 00:47:36.320
build a clock, but I learned so much doing that.

00:47:36.559 --> 00:47:38.460
What kind of clock did you build? It was like

00:47:38.460 --> 00:47:41.079
at camp and it was like a wooden it was like

00:47:41.079 --> 00:47:44.219
woodworking and then we put like the motor in

00:47:44.219 --> 00:47:46.719
and it was able it was able to do it yeah it

00:47:46.719 --> 00:47:49.659
was it was pretty cool I remember like staining

00:47:49.659 --> 00:47:51.980
the wood and you know cutting it and sanding

00:47:51.980 --> 00:47:53.960
it and doing all that stuff like no like do I

00:47:53.960 --> 00:47:56.579
ever need to do that stuff but it's like it's

00:47:56.579 --> 00:47:58.719
like being able to do it I think there's like

00:47:58.719 --> 00:48:02.079
um One of my friends in college always used to

00:48:02.079 --> 00:48:05.179
talk about this time that he was working as a

00:48:05.179 --> 00:48:08.159
farmer, basically, and how much it changed the

00:48:08.159 --> 00:48:10.360
way that he just thought about every like everything

00:48:10.360 --> 00:48:15.619
like what it takes to maintain a garden or a

00:48:15.619 --> 00:48:17.900
crop a set of crops. It's like, those are the

00:48:17.900 --> 00:48:20.360
lessons that are very important that you need

00:48:20.360 --> 00:48:23.369
when you're doing anything that has any anything

00:48:23.369 --> 00:48:25.670
that's not just like a one time thing right like

00:48:25.670 --> 00:48:28.130
which code very much is it's something that evolves

00:48:28.130 --> 00:48:30.210
over time so being able to maintain and take

00:48:30.210 --> 00:48:33.030
care of your code is is something that that's

00:48:33.030 --> 00:48:35.840
that's very important And then you also raise

00:48:35.840 --> 00:48:37.860
a really interesting point. It's like, yes, there

00:48:37.860 --> 00:48:40.800
is the democratization, but there's also the

00:48:40.800 --> 00:48:43.260
concentration of power. You know, whether you

00:48:43.260 --> 00:48:45.599
believe in like the 10 X engineer or not, but

00:48:45.599 --> 00:48:48.280
whatever, a very highly effective engineer that's

00:48:48.280 --> 00:48:50.840
then able to automate more of their work is just

00:48:50.840 --> 00:48:53.059
going to become extremely, extremely effective.

00:48:53.340 --> 00:48:54.980
Doesn't everyone believe in the 10 X engineer?

00:48:55.099 --> 00:48:56.980
Like, how could you not? Like, if you actually

00:48:56.980 --> 00:48:59.920
work with engineers, I think that's like. I mean.

00:49:00.320 --> 00:49:04.400
I don't know, I just didn't want to assume that

00:49:04.400 --> 00:49:06.559
you did, because some people are very much like,

00:49:06.639 --> 00:49:08.840
I don't know, I've seen a lot of polarity on

00:49:08.840 --> 00:49:11.840
that. Really? Yeah, some people really don't

00:49:11.840 --> 00:49:14.440
and then some people really are. Maybe you're

00:49:14.440 --> 00:49:16.699
right, maybe I'm just not talking to the right

00:49:16.699 --> 00:49:18.960
people. I think that if you know engineers, you

00:49:18.960 --> 00:49:20.940
know that there really are people that - I can't

00:49:20.940 --> 00:49:25.809
imagine working with engineers enough. No, 100%.

00:49:25.809 --> 00:49:27.789
No, there are people that are just capable of

00:49:27.789 --> 00:49:30.369
doing that are capable of architecting things

00:49:30.369 --> 00:49:33.429
that just other people are not capable of doing

00:49:33.429 --> 00:49:36.789
and make an impact that's just much larger on

00:49:36.789 --> 00:49:41.809
things 100%. Yeah. But yeah, so I just didn't

00:49:41.809 --> 00:49:43.650
want to assume that that's what you thought too.

00:49:45.409 --> 00:49:49.210
I'm moving into like advice and things like that.

00:49:49.210 --> 00:49:52.400
I'm going to switch this one. What advice would

00:49:52.400 --> 00:49:55.079
you give yourself when you were starting your

00:49:55.079 --> 00:50:00.019
career? Yeah, I think the most important advice

00:50:00.019 --> 00:50:01.860
I'd give myself is like, you know Lucas like

00:50:01.860 --> 00:50:04.420
you think you're late, but you're early like

00:50:04.420 --> 00:50:07.460
I just always felt bad like I you know, I Like

00:50:07.460 --> 00:50:11.940
it's funny. Like I I graduated, you know, like

00:50:11.940 --> 00:50:17.320
2005 and I got an offer to go to Google and I

00:50:17.320 --> 00:50:18.619
think I didn't take it honestly cuz I was like

00:50:18.619 --> 00:50:21.000
all my friends went there like You know, it's

00:50:21.000 --> 00:50:23.800
like, it must be like late. You know, I think,

00:50:23.820 --> 00:50:25.320
yeah, this is like my whole career is always

00:50:25.320 --> 00:50:28.460
like feeling bad that I was like late to stuff,

00:50:28.460 --> 00:50:31.840
but actually like, you know, I was like, um,

00:50:32.059 --> 00:50:35.519
like early to everything. And so, um, yeah, I

00:50:35.519 --> 00:50:37.880
don't know. Maybe that would make me just feel

00:50:37.880 --> 00:50:41.639
better about my career advice is tricky, right?

00:50:41.980 --> 00:50:44.300
Yeah. What advice would I give myself? It's probably

00:50:44.300 --> 00:50:46.119
something I just think about more. That's a good

00:50:46.119 --> 00:50:49.460
one though. And I think another thing. That I

00:50:49.460 --> 00:50:52.559
think another thing that I realized later Is

00:50:52.559 --> 00:50:54.679
it carving out like huge blocks of time to stay

00:50:54.679 --> 00:50:56.840
technical? I kind of felt like bad about that

00:50:56.840 --> 00:50:59.980
actually when I was like first You know for like

00:50:59.980 --> 00:51:02.619
a long time I felt like it was sort of like I

00:51:02.619 --> 00:51:04.719
you know I had these coaches like executive coaches

00:51:04.719 --> 00:51:06.840
like if you go off for a week You need to like

00:51:06.840 --> 00:51:08.840
come back with like something you produced and

00:51:08.840 --> 00:51:10.619
like a message to go like you can't just like

00:51:10.619 --> 00:51:12.219
Disappear for a week, and that's like you know

00:51:12.219 --> 00:51:14.880
fuck it like I disappear for a week Like if like,

00:51:15.059 --> 00:51:16.780
cause I would go in like paternity leave and

00:51:16.780 --> 00:51:19.039
I would come back with like so many ideas and

00:51:19.039 --> 00:51:20.480
like you can't really concentrate on paternity

00:51:20.480 --> 00:51:21.960
leave, you know, but it's like, I think like

00:51:21.960 --> 00:51:23.559
saying technical, it's going to like be like

00:51:23.559 --> 00:51:26.019
lots of ideas. Actually it was, I met the new

00:51:26.019 --> 00:51:29.639
relic CEO and he was running a public company

00:51:29.639 --> 00:51:31.340
at the time. And he was like, you know, I spend

00:51:31.340 --> 00:51:34.179
one week a month just like basically by myself

00:51:34.179 --> 00:51:35.519
or with the smaller people just like writing

00:51:35.519 --> 00:51:37.400
code. And that like, it was pretty inspiring.

00:51:38.079 --> 00:51:39.760
And I think another thing it's like, you know,

00:51:39.760 --> 00:51:42.519
I think when I I think for a long time, people

00:51:42.519 --> 00:51:45.519
have this stupid idea that you can have these

00:51:45.519 --> 00:51:49.039
executives that don't stay technical or able

00:51:49.039 --> 00:51:53.300
to do the IC work. Because management is its

00:51:53.300 --> 00:51:56.699
own skill. And so you should hire good managers,

00:51:56.780 --> 00:52:00.260
not good ICs. And I just think that's so stupid.

00:52:00.500 --> 00:52:03.639
How many times do you have to see you hire someone

00:52:03.639 --> 00:52:06.960
who seems like a good manager, but they can't

00:52:06.960 --> 00:52:09.610
do the individual work and watch them fail? Before

00:52:09.610 --> 00:52:12.070
I said, give me a fucking break. Like, you know,

00:52:12.489 --> 00:52:13.929
like, you know, if you're going to like work

00:52:13.929 --> 00:52:15.969
for me, you better be able to do the IC job.

00:52:16.849 --> 00:52:19.230
And like, I do not know how the company's function

00:52:19.230 --> 00:52:21.590
without that mindset. I'm just like baffled.

00:52:21.690 --> 00:52:23.530
I thought maybe it was just like me, but I actually

00:52:23.530 --> 00:52:25.349
just think there's like a lot of bad executives

00:52:25.349 --> 00:52:26.730
that just kind of like keep their head down and

00:52:26.730 --> 00:52:29.210
somehow, you know, keep like shuffling around

00:52:29.210 --> 00:52:31.929
and like, you see like VPs of engineering that

00:52:31.929 --> 00:52:33.590
are pissed if you like ask them to like code

00:52:33.590 --> 00:52:37.260
in an interview. It's come on. I'd be like. Excited

00:52:37.260 --> 00:52:41.260
if you give me like a like a leak code Google

00:52:41.260 --> 00:52:44.039
code interview I might not be like awesome at

00:52:44.039 --> 00:52:47.400
it, but I would be like pumped Yeah Corvish has

00:52:47.400 --> 00:52:53.900
done that when they when they hired me Go back

00:52:53.900 --> 00:52:56.380
go back to the due diligence and tell me you

00:52:56.380 --> 00:52:59.940
want you want me code See you better be able

00:52:59.940 --> 00:53:06.269
to do the leak code I love it. I love it Yeah,

00:53:06.269 --> 00:53:10.650
it's it's huge. I mean getting shit done I mean

00:53:10.650 --> 00:53:13.389
like that's what it comes down to is You can

00:53:13.389 --> 00:53:15.389
talk about it and you can plan and you can do

00:53:15.389 --> 00:53:17.409
whatever you want But it's like are you able

00:53:17.409 --> 00:53:19.510
to actually execute and are you able to actually

00:53:19.510 --> 00:53:22.050
get this done? And the amounts that you can get

00:53:22.050 --> 00:53:24.329
done in a short amount of focus time is like

00:53:24.329 --> 00:53:26.670
is unbelievable So you better have people around

00:53:26.670 --> 00:53:29.329
you that are able to do you know do do things?

00:53:29.949 --> 00:53:33.090
I think that in what I love about startups is

00:53:33.090 --> 00:53:35.150
that you don't have the time, you don't have

00:53:35.150 --> 00:53:39.650
the fact, you don't have that, you can't, you

00:53:39.650 --> 00:53:42.110
won't survive, right? If you're someone that

00:53:42.110 --> 00:53:43.929
can't execute, if you're somebody that's not

00:53:43.929 --> 00:53:46.130
able to get the job done, it's just like you

00:53:46.130 --> 00:53:48.789
don't have a, you just can't really have a role

00:53:48.789 --> 00:53:54.130
at a startup. That's what I love about entrepreneurship.

00:53:54.730 --> 00:53:56.510
Yeah, and thank you for the real answer. That

00:53:56.510 --> 00:54:00.769
was great. I like that a lot. So getting into

00:54:00.769 --> 00:54:03.030
just, you know, this is learning from machine

00:54:03.030 --> 00:54:06.510
learning. So I have to ask the question. What

00:54:06.510 --> 00:54:09.190
is a career in machine learning taught you about

00:54:09.190 --> 00:54:16.469
life? Well, I think like one perspective that

00:54:16.469 --> 00:54:18.289
I have from the machine or whether you think

00:54:18.289 --> 00:54:19.949
about a lot like when you do machine learning.

00:54:21.469 --> 00:54:22.949
This might not be true now. I think most of my

00:54:22.949 --> 00:54:25.250
machine learning was done like building and fine

00:54:25.250 --> 00:54:27.619
tuning models. This is not like the new, uh,

00:54:28.079 --> 00:54:30.239
you know, engineer stuff. But like, you know,

00:54:30.239 --> 00:54:34.940
I think when you're like training models, you,

00:54:34.940 --> 00:54:38.599
you kind of don't know, like when you should

00:54:38.599 --> 00:54:40.280
stop, like when something's like not working,

00:54:40.360 --> 00:54:42.760
like when you have, you know, an idea. So I think

00:54:42.760 --> 00:54:44.619
a lot about, okay, like when do we like pull

00:54:44.619 --> 00:54:47.320
the plug, you know, on this thing that I'm doing,

00:54:47.400 --> 00:54:49.159
like how much information do I like really need,

00:54:49.159 --> 00:54:51.880
you know? Um, and I think another thing is like

00:54:51.880 --> 00:54:54.719
the, the feedback loops are kind of like your

00:54:54.719 --> 00:54:58.539
unit of um, your unit of work. So I think like

00:54:58.539 --> 00:55:00.340
the faster you're like getting feedback and the

00:55:00.340 --> 00:55:02.599
more feedback that you're getting, I'm sort of

00:55:02.599 --> 00:55:05.920
obsessed with getting, you know, feedback, whatever

00:55:05.920 --> 00:55:07.739
that means, you know, for whatever I'm doing,

00:55:07.980 --> 00:55:09.880
kind of getting that like quickly to try to get,

00:55:09.880 --> 00:55:11.920
you know, better, whatever that thing is. I think

00:55:11.920 --> 00:55:13.320
that kind of comes from doing machine learning

00:55:13.320 --> 00:55:15.340
where you're just trying to get like the information

00:55:15.340 --> 00:55:16.699
on like, is this working or not? So I can like

00:55:16.699 --> 00:55:18.980
move on to the next experiment as fast as I can.

00:55:19.989 --> 00:55:22.030
Yeah, that's interest that that's interesting

00:55:22.030 --> 00:55:25.469
so going to the first one where you're saying

00:55:25.469 --> 00:55:27.809
like knowing when to stop are you particularly

00:55:27.809 --> 00:55:30.769
talking about like you're training a model and

00:55:30.769 --> 00:55:33.550
you're Iterating on it and you're like when is

00:55:33.550 --> 00:55:36.190
it good enough to go into production or like?

00:55:36.849 --> 00:55:39.190
Like I was thinking of it's like okay. You have

00:55:39.190 --> 00:55:41.389
a model There's usually like an infinite number

00:55:41.389 --> 00:55:42.889
of things you could try to make it better like

00:55:42.889 --> 00:55:44.630
get more training data different types of hyper

00:55:44.630 --> 00:55:47.059
parameters different architecture, you know,

00:55:47.059 --> 00:55:48.159
all these different things, you know, and you're

00:55:48.159 --> 00:55:49.880
kind of like, okay, like, what should I try?

00:55:50.380 --> 00:55:52.039
And then when should I like move on to trying

00:55:52.039 --> 00:55:55.519
like the next thing? And I think that's, I mean,

00:55:55.519 --> 00:55:56.800
it's funny, I was starting to say, I think that's

00:55:56.800 --> 00:55:58.300
like effected my life, but I think only the fact

00:55:58.300 --> 00:56:00.599
that I think about that a lot kind of comes from

00:56:00.599 --> 00:56:02.280
in machine learning, like think about a lot.

00:56:02.280 --> 00:56:04.119
And it's like, it's hard, you never really get

00:56:04.119 --> 00:56:06.400
that feedback on if you like cut that experiment

00:56:06.400 --> 00:56:08.699
short fast enough. And I think different than

00:56:08.699 --> 00:56:12.699
like more like statistical practices, you kind

00:56:12.699 --> 00:56:15.500
of need to make these decisions. within perfect

00:56:15.500 --> 00:56:18.280
information, right? Which is more like life.

00:56:18.900 --> 00:56:20.760
You know, like, I think like medical stuff, I

00:56:20.760 --> 00:56:23.800
felt very familiar having like medical issues

00:56:23.800 --> 00:56:26.000
where I'm trying to look at a bunch of studies

00:56:26.000 --> 00:56:28.860
that don't kind of line up and none of them are

00:56:28.860 --> 00:56:30.340
like seem totally statistically significant,

00:56:30.480 --> 00:56:32.139
but yet you have to make a decision about your

00:56:32.139 --> 00:56:34.539
health that's important. That felt like a very

00:56:34.539 --> 00:56:37.860
familiar situation to me from machine learning

00:56:37.860 --> 00:56:40.130
where, you know, you've done a bunch of tests,

00:56:40.289 --> 00:56:41.809
none of them are perfect. Probably there are

00:56:41.809 --> 00:56:45.909
some bugs in your code one time, so it's questionable

00:56:45.909 --> 00:56:51.730
if you can trust it. So you have 30 messy data

00:56:51.730 --> 00:56:54.769
points, and then you have to decide what information

00:56:54.769 --> 00:56:57.929
to collect next or what to try next. I don't

00:56:57.929 --> 00:56:59.989
know. I think that's a common situation in life.

00:57:00.590 --> 00:57:04.730
I think machine learning gives you a lot of practice

00:57:04.730 --> 00:57:10.639
handling that. But I'm not sure I would claim

00:57:10.639 --> 00:57:12.980
to be good at it. I just, I don't know. I think

00:57:12.980 --> 00:57:15.880
I just, you know, you counter that a lot to get

00:57:15.880 --> 00:57:17.340
that perspective. And you know, like running

00:57:17.340 --> 00:57:18.579
a company, it's like, okay, you know, you try

00:57:18.579 --> 00:57:21.159
some like new growth. Like, it's funny, our head

00:57:21.159 --> 00:57:22.820
of growth, it was actually, she was a machine

00:57:22.820 --> 00:57:24.079
learning person. And it was like interesting

00:57:24.079 --> 00:57:27.059
to watch her. You're kind of thinking that it

00:57:27.059 --> 00:57:29.880
was like - Lavanya? Lavanya, yeah, yeah. So her

00:57:29.880 --> 00:57:33.179
growth stuff felt like machine learning was,

00:57:33.280 --> 00:57:34.820
okay, we're going to run these like 10 experiments.

00:57:35.159 --> 00:57:36.679
you know we're going to try to like cut them

00:57:36.679 --> 00:57:39.219
off with like imperfect information get the like

00:57:39.219 --> 00:57:41.599
11th experiment in there it kind of made me realize

00:57:41.599 --> 00:57:44.119
it's actually a really similar thing to trying

00:57:44.119 --> 00:57:46.579
to find the best um you know ml model for some

00:57:46.579 --> 00:57:49.920
task yeah that's what i was going to say is that

00:57:49.920 --> 00:57:52.400
it while in machine learning, you're dealing

00:57:52.400 --> 00:57:54.460
with uncertainty, also in entrepreneurship, you're

00:57:54.460 --> 00:57:57.019
dealing with uncertainty and making decisions,

00:57:57.019 --> 00:57:59.539
yeah, with imperfect information. And you can

00:57:59.539 --> 00:58:02.199
just make the best decision, you know, in that

00:58:02.199 --> 00:58:05.780
moment. And then going into your your other point,

00:58:05.840 --> 00:58:09.099
yeah, about feedback, it's, yeah, I mean, it's

00:58:09.099 --> 00:58:11.559
it's crucial being able to collect the right

00:58:11.559 --> 00:58:13.260
feedback, being able to understand how people

00:58:13.260 --> 00:58:15.219
are actually using the thing that you created,

00:58:15.219 --> 00:58:18.139
and then trying to incorporate that back into

00:58:18.139 --> 00:58:21.210
the model, whether it's some Sometimes it's like,

00:58:21.210 --> 00:58:23.530
yeah, obviously, you could change the data set

00:58:23.530 --> 00:58:25.610
that you train the model on, or you could change

00:58:25.610 --> 00:58:28.730
the way that the user interacts with your model

00:58:28.730 --> 00:58:32.409
and things like that. That's great. Is there

00:58:32.409 --> 00:58:35.809
anything else that you'd want to share about

00:58:35.809 --> 00:58:42.150
that? Well, I don't know. I guess I was thinking

00:58:42.150 --> 00:58:46.630
about you had an earlier question about how do

00:58:46.630 --> 00:58:49.110
you... like what advice would you give yourself?

00:58:49.610 --> 00:58:51.710
And I think like one of the funny things that

00:58:51.710 --> 00:58:54.869
I've experienced in life is that you get a lot

00:58:54.869 --> 00:58:57.349
of advice that's like really good and you just

00:58:57.349 --> 00:59:01.610
don't like do it enough, you know, and sort of

00:59:01.610 --> 00:59:04.170
like over time you like learn to just like do

00:59:04.170 --> 00:59:05.690
that like obvious thing that you'd obviously

00:59:05.690 --> 00:59:08.150
like tell yourself just like even to a more bigger

00:59:08.150 --> 00:59:10.849
degree than um than you're getting. It's like

00:59:10.849 --> 00:59:12.909
I feel like asking customers for feedback is

00:59:12.909 --> 00:59:15.699
such a funny one where like You know, just no

00:59:15.699 --> 00:59:19.079
one, no entrepreneur does it enough. And it's

00:59:19.079 --> 00:59:21.239
like, it's not the only thing, right? Like, you

00:59:21.239 --> 00:59:23.079
know, like, I mean, just sort of blindly doing

00:59:23.079 --> 00:59:26.239
what customers want is kind of a stupid way,

00:59:26.239 --> 00:59:29.400
you know, to operate. But I think like understanding

00:59:29.400 --> 00:59:31.340
customers deeply and asking them like what they're

00:59:31.340 --> 00:59:33.500
thinking and like, you know, like what they would

00:59:33.500 --> 00:59:35.500
want. It's like, I really think that that, I

00:59:35.500 --> 00:59:38.219
think all my co -founders, we really, like what

00:59:38.219 --> 00:59:39.940
I think about like all my co -founders, my two

00:59:39.940 --> 00:59:41.420
other co -founders, we all really had in common

00:59:41.420 --> 00:59:44.219
was. We've really felt like super bad when customers

00:59:44.219 --> 00:59:48.119
were unhappy and we like worried about it a lot.

00:59:48.500 --> 00:59:51.079
And it would, it really like hurt our feelings.

00:59:51.320 --> 00:59:52.679
You know what I mean? And so like, we would,

00:59:53.400 --> 00:59:55.260
I feel like we all kind of just like maybe a

00:59:55.260 --> 00:59:56.599
little bit of like an anxious style. Like we'd

00:59:56.599 --> 00:59:59.780
be like asking her feedback. Cause I think we'd

00:59:59.780 --> 01:00:02.500
be just like worrying like, okay, like, is there

01:00:02.500 --> 01:00:04.119
some problem? You know, like why aren't you using

01:00:04.119 --> 01:00:07.500
our product? And I think like getting that feedback

01:00:07.500 --> 01:00:12.739
from customers like all the time. you know, just

01:00:12.739 --> 01:00:15.579
like really, really helped us be successful.

01:00:15.579 --> 01:00:18.099
And it's so basic, but it's like, I just, I know

01:00:18.099 --> 01:00:20.139
I like watch a lot of entrepreneurs like not

01:00:20.139 --> 01:00:22.659
do that. And I think even in my first company,

01:00:22.699 --> 01:00:25.780
I probably didn't do it enough. Um, honestly,

01:00:25.780 --> 01:00:26.860
as I'm talking about this, probably should do

01:00:26.860 --> 01:00:29.960
it even more, um, you know, now, but it's funny

01:00:29.960 --> 01:00:31.360
because everyone knows you're supposed to do

01:00:31.360 --> 01:00:33.860
that, but I think it's just, it's like pretty

01:00:33.860 --> 01:00:36.039
annoying to do it. It's like a little bit stressful,

01:00:36.039 --> 01:00:39.480
you know, um, you know, we just don't do it enough.

01:00:39.760 --> 01:00:42.800
Yeah, well, you make a really good point in Well,

01:00:42.940 --> 01:00:44.960
you made many good points, but the the point

01:00:44.960 --> 01:00:48.300
around like you don't just blindly take customer

01:00:48.300 --> 01:00:51.559
feedback and then apply it. It's the step of

01:00:52.510 --> 01:00:55.550
deeply understanding the pain and the problem

01:00:55.550 --> 01:00:59.210
and the need of your customers to then inform

01:00:59.210 --> 01:01:01.949
your roadmap and then to inform the decisions

01:01:01.949 --> 01:01:04.849
that the decisions that you're making. And I

01:01:04.849 --> 01:01:07.570
think that that's where the challenge is, is

01:01:07.570 --> 01:01:10.210
that you can get tons of feedback from your customers.

01:01:10.469 --> 01:01:13.389
But the hard thing is to really have the empathy

01:01:13.389 --> 01:01:15.429
to put yourself in the shoes of your customers

01:01:15.429 --> 01:01:17.610
to understand what their problems are what the

01:01:17.610 --> 01:01:20.340
incentives that they're dealing with are. I mean,

01:01:20.579 --> 01:01:23.420
I know that from my perspective, I think about

01:01:23.420 --> 01:01:26.460
the problems that I'm trying to solve, and they're

01:01:26.460 --> 01:01:30.360
often not the problems that my people who are

01:01:30.360 --> 01:01:32.119
going to pay for the software that I'm creating

01:01:32.119 --> 01:01:35.440
are going to are dealing with. And it's that

01:01:35.440 --> 01:01:37.980
gap that makes that such a hard, such a hard,

01:01:38.199 --> 01:01:42.150
such a hard thing to do. Yeah, totally. Lucas,

01:01:42.150 --> 01:01:44.469
this has been like just like unbelievable. I

01:01:44.469 --> 01:01:46.409
said to you offline, but you're one of the people

01:01:46.409 --> 01:01:49.309
that I wanted to get on this podcast for so long.

01:01:49.590 --> 01:01:52.150
Yeah, it's it's incredible. Maybe you think that

01:01:52.150 --> 01:01:54.550
you were late or whatever. You were early to

01:01:54.550 --> 01:01:58.210
the game. You know, you've had such an incredible

01:01:58.210 --> 01:02:02.369
career to successful exits, not exits, but you

01:02:02.369 --> 01:02:04.590
know what I mean in terms of your your startups.

01:02:05.030 --> 01:02:08.170
It's it's really, yeah, it's really admirable.

01:02:08.389 --> 01:02:11.530
It's it's it's really admirable. And also just

01:02:11.769 --> 01:02:15.429
the type of person that you are. It was so amazing

01:02:15.429 --> 01:02:18.550
getting to know you a little bit at Fully Connected

01:02:18.550 --> 01:02:22.630
and just how real you are as a person and not

01:02:22.630 --> 01:02:28.050
the usual CEO. And I really just appreciate you,

01:02:28.190 --> 01:02:29.889
so many of the things that you're doing, the

01:02:29.889 --> 01:02:32.550
incredible software that you're doing, but just

01:02:32.550 --> 01:02:36.610
the force that you are in this field. So thank

01:02:36.610 --> 01:02:39.309
you so much. It's really amazing to have the

01:02:39.309 --> 01:02:41.250
opportunity to talk with you. No, thanks, Ed.

01:02:41.449 --> 01:02:43.730
Really appreciate it. Yeah. Is there anywhere

01:02:43.730 --> 01:02:47.670
where listeners can learn more about you? Well,

01:02:47.670 --> 01:02:50.369
just go to WMB .com. But if it's before Fully

01:02:50.369 --> 01:02:52.809
Connected, then they should come to Fully Connected.

01:02:52.869 --> 01:02:55.010
We'd love to have your listeners at our conference.

01:02:56.259 --> 01:02:59.760
It's a great conference. I'm in New York. I don't

01:02:59.760 --> 01:03:01.739
know if I'll be able to make the trip to California

01:03:01.739 --> 01:03:03.900
this year, but I'm sure I'll be there again in

01:03:03.900 --> 01:03:07.239
the future. I would encourage people to come.

01:03:07.500 --> 01:03:10.579
Lucas, man, what a pleasure. Thank you so much.

01:03:10.980 --> 01:03:17.820
I really appreciate your time. Thanks, man. On

01:03:17.820 --> 01:03:20.079
this episode of Learning from Machine Learning,

01:03:20.539 --> 01:03:23.159
I had the privilege of speaking with Lucas Biewald,

01:03:23.659 --> 01:03:27.119
co -founder and CEO of Weights and Biases. We

01:03:27.119 --> 01:03:29.360
trace this journey from programming games as

01:03:29.360 --> 01:03:31.980
a kid to building one of the most essential tools

01:03:31.980 --> 01:03:36.119
in AI development today. Lucas's career demonstrates

01:03:36.119 --> 01:03:39.699
that conviction often matters more than consensus.

01:03:40.400 --> 01:03:43.860
From surviving the AI winter in the mid -2000s

01:03:43.860 --> 01:03:46.099
to the AlphaGo moment that changed everything.

01:03:46.409 --> 01:03:49.929
Lucas reminds us that we're still early in humanity's

01:03:49.929 --> 01:03:53.670
most transformative project. He challenges conventional

01:03:53.670 --> 01:03:56.969
leadership wisdom and shares his unwavering belief

01:03:56.969 --> 01:04:00.030
that executives must stay technical. He explains

01:04:00.030 --> 01:04:02.510
bluntly, if you're going to work for me, you

01:04:02.510 --> 01:04:06.590
better be able to do the IC job. Most importantly,

01:04:07.090 --> 01:04:10.170
Lucas's philosophy that feedback loops are your

01:04:10.170 --> 01:04:13.269
unit of work transforms how we approach both

01:04:13.269 --> 01:04:16.369
machine learning and life. His advice to his

01:04:16.369 --> 01:04:19.190
younger self cuts through common doubts. You

01:04:19.190 --> 01:04:22.769
think you're late, but you're early. In a world

01:04:22.769 --> 01:04:25.630
racing towards progress by any means necessary,

01:04:26.269 --> 01:04:29.449
this reminder couldn't be more relevant. Thank

01:04:29.449 --> 01:04:32.329
you for listening. Be sure to subscribe and share

01:04:32.329 --> 01:04:35.530
with a friend or colleague. Until next time,

01:04:36.110 --> 01:04:36.989
keep on learning.