1 00:00:00,000 --> 00:00:06,160 But Tara, you love Jeopardy, right? 2 00:00:06,160 --> 00:00:07,920 I do love Jeopardy. 3 00:00:07,920 --> 00:00:10,240 Would you want to be a Jeopardy? 4 00:00:10,240 --> 00:00:17,600 Yes, but Willa Fortune is the top of my game show host thing that it's not really a host 5 00:00:17,600 --> 00:00:21,960 that I want to be on because I grew up watching it with my grandmother and I still watch it 6 00:00:21,960 --> 00:00:26,400 to this day with my boys and it's our thing every night that we do that together. 7 00:00:26,400 --> 00:00:30,720 So Willa Fortune has my heart right now. 8 00:00:30,720 --> 00:00:31,720 That's awesome. 9 00:00:31,720 --> 00:00:32,720 Love that. 10 00:00:32,720 --> 00:00:37,840 Matthew, if you had to be on a game show, what would it be? 11 00:00:37,840 --> 00:00:43,600 I'd not, practically. 12 00:00:43,600 --> 00:00:45,100 Willa Fortune's good. 13 00:00:45,100 --> 00:00:49,240 Jeopardy would be fun, but I think Jeopardy's one of those ones where the second you get 14 00:00:49,240 --> 00:00:53,880 up there you'd forget half the things you know. 15 00:00:53,880 --> 00:01:00,920 So I don't know which ones I'd be good at, but the idea of doing the amazing race has 16 00:01:00,920 --> 00:01:03,320 always seemed really fun. 17 00:01:03,320 --> 00:01:04,320 So yeah. 18 00:01:04,320 --> 00:01:06,320 I like it. 19 00:01:06,320 --> 00:01:07,320 I like it. 20 00:01:07,320 --> 00:01:08,320 Todd. 21 00:01:08,320 --> 00:01:09,320 Family Feud. 22 00:01:09,320 --> 00:01:11,320 It would be my opportunity for my stand-up routine. 23 00:01:11,320 --> 00:01:12,320 You took my. 24 00:01:12,320 --> 00:01:15,920 You're being the same team because that was going to be mine. 25 00:01:15,920 --> 00:01:16,920 So true. 26 00:01:16,920 --> 00:01:22,800 Coming soon to a podcast near you, Family Feud is the idea edition. 27 00:01:22,800 --> 00:01:24,280 Family marketing versus security. 28 00:01:24,280 --> 00:01:25,280 It'll be great. 29 00:01:25,280 --> 00:01:26,280 It'll be great. 30 00:01:26,280 --> 00:01:28,600 What about you, Nate? 31 00:01:28,600 --> 00:01:34,280 I totally agree with Matthew of the amazing race. 32 00:01:34,280 --> 00:01:39,200 So I watched the first season and I really enjoy how those ones are so drawn out. 33 00:01:39,200 --> 00:01:42,760 They didn't keep everyone right next to each other. 34 00:01:42,760 --> 00:01:44,960 Marketing did ask which one do I think I could win? 35 00:01:44,960 --> 00:01:49,800 I'm ahead of this and I did say the bachelor because I'd be the only guy making it to the 36 00:01:49,800 --> 00:01:55,800 end, although I would not want to be on the show. 37 00:01:55,800 --> 00:01:56,800 I know I'm stepping back in. 38 00:01:56,800 --> 00:01:59,000 And I am married. 39 00:01:59,000 --> 00:02:02,600 You've just reminded me there's a game show. 40 00:02:02,600 --> 00:02:03,640 I suppose you call it a game show. 41 00:02:03,640 --> 00:02:10,520 I watch called Um Actually that you can find a couple episodes off on YouTube that is basically 42 00:02:10,520 --> 00:02:17,720 just nerd stuff that is incredibly fun if you're into that. 43 00:02:17,720 --> 00:02:21,640 I think I would have a lot of fun playing. 44 00:02:21,640 --> 00:02:22,640 That's awesome. 45 00:02:22,640 --> 00:02:25,520 Now that everybody's stopped listening to the podcast to go on YouTube, pause that for 46 00:02:25,520 --> 00:02:27,520 a second and come back to the podcast. 47 00:02:27,520 --> 00:02:28,520 That sounds awesome. 48 00:02:28,520 --> 00:02:29,920 What about you, Ariel? 49 00:02:29,920 --> 00:02:30,920 That's great. 50 00:02:30,920 --> 00:02:33,920 Oh, I don't know. 51 00:02:33,920 --> 00:02:37,800 I don't know which one I would win, but I think I always wanted to try. 52 00:02:37,800 --> 00:02:41,000 Is it a game show that like Ninja Warrior? 53 00:02:41,000 --> 00:02:42,920 I just think that that's cool. 54 00:02:42,920 --> 00:02:45,320 I just want to climb on all that stuff. 55 00:02:45,320 --> 00:02:47,920 I would be out immediately though. 56 00:02:47,920 --> 00:02:50,040 I would not make it far. 57 00:02:50,040 --> 00:02:51,040 Just the experience. 58 00:02:51,040 --> 00:02:54,320 Make it past like the first thing fall in the water and you did it. 59 00:02:54,320 --> 00:02:56,320 Yeah, for sure. 60 00:02:56,320 --> 00:02:57,320 Yes. 61 00:02:57,320 --> 00:02:58,320 Oh man. 62 00:02:58,320 --> 00:02:59,320 Nice. 63 00:02:59,320 --> 00:03:01,320 Kelsey, you answered yours already, right? 64 00:03:01,320 --> 00:03:02,320 I did. 65 00:03:02,320 --> 00:03:03,320 You guys are going on Family Feud. 66 00:03:03,320 --> 00:03:04,800 Yeah, I piggybacked here. 67 00:03:04,800 --> 00:03:08,320 I was already did like lead up to it and then I was like, well, Todd took it. 68 00:03:08,320 --> 00:03:10,320 So let's say it's over. 69 00:03:10,320 --> 00:03:11,320 It's over. 70 00:03:11,320 --> 00:03:12,320 But I agree with you, Ariel. 71 00:03:12,320 --> 00:03:15,320 I think that would be really fun to do like Ninja Warrior. 72 00:03:15,320 --> 00:03:20,920 And then I'm sometimes when I'm watching that, my partner will, I'll be like, oh, I think 73 00:03:20,920 --> 00:03:21,920 I could do that one. 74 00:03:21,920 --> 00:03:26,720 And my partner has to remind me that it's been a long time since I did rock climbing 75 00:03:26,720 --> 00:03:27,720 of any significance. 76 00:03:27,720 --> 00:03:32,320 The one that will get me is the pull up bar where you have to like jump. 77 00:03:32,320 --> 00:03:33,320 The salmon one. 78 00:03:33,320 --> 00:03:34,320 Yeah, the salmon one. 79 00:03:34,320 --> 00:03:37,320 That one I would be done on. 80 00:03:37,320 --> 00:03:38,320 Yeah. 81 00:03:38,320 --> 00:03:39,320 Oh yeah. 82 00:03:39,320 --> 00:03:41,040 They should just let us skip some. 83 00:03:41,040 --> 00:03:44,760 Like just let me try out some of them in a different order. 84 00:03:44,760 --> 00:03:46,560 And maybe I could, I could get through one or two. 85 00:03:46,560 --> 00:03:49,120 There is no participation trophy here. 86 00:03:49,120 --> 00:03:50,120 So. 87 00:03:50,120 --> 00:03:51,120 Yeah. 88 00:03:51,120 --> 00:03:55,320 Falling in the water feels like that's the participation trophy. 89 00:03:55,320 --> 00:03:56,320 Yeah. 90 00:03:56,320 --> 00:03:57,320 Love it. 91 00:03:57,320 --> 00:03:58,320 Awesome. 92 00:03:58,320 --> 00:04:08,520 Well, I guess speaking, you know, of a human's shortcomings is a little bit what we're going 93 00:04:08,520 --> 00:04:09,760 to talk about today. 94 00:04:09,760 --> 00:04:12,200 And our tech for business podcast. 95 00:04:12,200 --> 00:04:15,040 And thank you. 96 00:04:15,040 --> 00:04:22,280 Our joined by our daughter, CEO and CISO Nate, our director of cybersecurity and Matthew 97 00:04:22,280 --> 00:04:23,280 RBC. 98 00:04:23,280 --> 00:04:30,480 So and today we're posing the question, do we trust deep learning too much? 99 00:04:30,480 --> 00:04:34,520 And I know before the podcast, Nate was kind of talking a little bit about what that is. 100 00:04:34,520 --> 00:04:38,720 And I think we're going to start there is what is deep learning? 101 00:04:38,720 --> 00:04:43,880 And I don't know if you could give us maybe an example so we can kind of connect it into 102 00:04:43,880 --> 00:04:46,080 the real world a little bit more. 103 00:04:46,080 --> 00:04:47,080 What is it? 104 00:04:47,080 --> 00:04:48,080 Yeah. 105 00:04:48,080 --> 00:04:51,840 So I promise I won't go too technical. 106 00:04:51,840 --> 00:04:58,720 But it's a subset of artificial intelligence, a subset of machine learning. 107 00:04:58,720 --> 00:05:03,280 It's just a very. 108 00:05:03,280 --> 00:05:10,120 Big algorithms that traditional machine learning models couldn't address. 109 00:05:10,120 --> 00:05:13,440 And so there's a great YouTuber. 110 00:05:13,440 --> 00:05:16,600 He's the best speaker I've ever seen in person. 111 00:05:16,600 --> 00:05:22,000 I highly recommend looking him up at some point, but Tanmay Bakshi, he has a YouTube 112 00:05:22,000 --> 00:05:24,880 channel, Tanmay teaches. 113 00:05:24,880 --> 00:05:28,840 He's been all around the world speaking on machine learning. 114 00:05:28,840 --> 00:05:33,640 He's 19 years old, phenomenal, wildly smart guy. 115 00:05:33,640 --> 00:05:39,000 Anyways, he has a lot of stuff where and the last time I saw him in person, he said artificial 116 00:05:39,000 --> 00:05:42,360 intelligence is a user experience. 117 00:05:42,360 --> 00:05:49,800 It's kind of what we perceive is these machines seem intelligent. 118 00:05:49,800 --> 00:05:55,480 And then machine learning is the mathematics and algorithms that go beneath that. 119 00:05:55,480 --> 00:06:02,200 And then deep learning is a method of machine learning that says there's new algorithms 120 00:06:02,200 --> 00:06:10,920 to build relationships between databases or data that you wouldn't typically have. 121 00:06:10,920 --> 00:06:18,520 Math and metal math equations between the two so it can make better, more accurate predictions. 122 00:06:18,520 --> 00:06:24,640 And then that's where we start to see this really start coming into the fruition of new 123 00:06:24,640 --> 00:06:36,160 technologies and the fruition of new methodologies to better predict really the outcome of whatever 124 00:06:36,160 --> 00:06:37,160 we're trying to evaluate. 125 00:06:37,160 --> 00:06:45,360 It could be cybersecurity, healthcare, whatever it is, but it's just a newer approach to it, 126 00:06:45,360 --> 00:06:49,000 and it's using just different algorithms. 127 00:06:49,000 --> 00:06:55,960 So I was talking with someone the other day about analogies to kind of get through all 128 00:06:55,960 --> 00:07:01,640 three and I like your user experience description of how we interact with them. 129 00:07:01,640 --> 00:07:09,800 But in terms of trying to visualize what this really looks like, for me at least AI is the 130 00:07:09,800 --> 00:07:16,840 ability for a machine to solve long division without us having to do anything. 131 00:07:16,840 --> 00:07:24,600 Machine learning is it learning how to, like when to do those equations versus other equations. 132 00:07:24,600 --> 00:07:29,320 And then deep learning is knowing how to manipulate the equation so well that it can solve other 133 00:07:29,320 --> 00:07:34,440 problems with that initial bit of information. 134 00:07:34,440 --> 00:07:41,640 So it's things that we kind of do expanded schemas is kind of how it works for us. 135 00:07:41,640 --> 00:07:45,160 We have plans and how we do things and then sometimes something breaks that and so we 136 00:07:45,160 --> 00:07:47,360 have to work on the fly. 137 00:07:47,360 --> 00:07:49,560 Computers obviously can't do that. 138 00:07:49,560 --> 00:07:56,840 Deep learning is about giving it enough data that it can do that to an extent with some 139 00:07:56,840 --> 00:08:00,680 calculations and everything else that go into that. 140 00:08:00,680 --> 00:08:01,680 Yeah. 141 00:08:01,680 --> 00:08:07,080 So I'm just going to quick reference that Tanmay, Bakshi one more time and then I'm done, 142 00:08:07,080 --> 00:08:08,360 I promise. 143 00:08:08,360 --> 00:08:14,480 But one of the illustrations that he gave was machine learning is taking the defined 144 00:08:14,480 --> 00:08:20,280 mathematical models and saying, here's how we would calculate the difference between 145 00:08:20,280 --> 00:08:22,440 a dog and a cat. 146 00:08:22,440 --> 00:08:24,640 We can't really do that today. 147 00:08:24,640 --> 00:08:30,560 Deep learning is going to try and figure that out saying what similarities, what math can 148 00:08:30,560 --> 00:08:35,800 we put in place to calculate how similar these two are. 149 00:08:35,800 --> 00:08:39,800 And we just input the different characteristics of it and it starts building all that data 150 00:08:39,800 --> 00:08:41,800 together. 151 00:08:41,800 --> 00:08:48,800 We didn't have to teach it that. 152 00:08:48,800 --> 00:08:50,680 Very oversimplified, I promise. 153 00:08:50,680 --> 00:08:59,440 Yeah, there might be some deep learning engineers who are raging a little bit at our simplifications 154 00:08:59,440 --> 00:09:00,440 here. 155 00:09:00,440 --> 00:09:04,480 But considering we're only seeing the outputs of them, we're not actually getting in there 156 00:09:04,480 --> 00:09:08,560 and coding it, I feel it's fair. 157 00:09:08,560 --> 00:09:13,120 We also have a lot of things that work in that machine learning side of things that are 158 00:09:13,120 --> 00:09:20,240 very prominent in cybersecurity, such as a lot of EDR software runs at that machine learning 159 00:09:20,240 --> 00:09:28,720 level and they're working on how to tie it into that deep learning side of things. 160 00:09:28,720 --> 00:09:37,440 One thing I want to mention, just because it's, I like to point to negatives directly, 161 00:09:37,440 --> 00:09:40,480 is a problem that we see all the time in tech. 162 00:09:40,480 --> 00:09:45,280 And it's not so much a problem as it is a form of, I don't want to say ignorance, because 163 00:09:45,280 --> 00:09:53,840 that seems a little too harsh, but it is just a lack of awareness really around where our 164 00:09:53,840 --> 00:09:55,360 failings are. 165 00:09:55,360 --> 00:09:58,160 So the whole point of deep learning is that we're feeding it data. 166 00:09:58,160 --> 00:10:02,080 We're feeding it data that we have so that it can learn new things and make decisions 167 00:10:02,080 --> 00:10:03,680 based on that data. 168 00:10:03,680 --> 00:10:11,560 And one of the best bits of information as a reminder of this that I learned while going 169 00:10:11,560 --> 00:10:20,720 through my degree was about research that was done on rodents and basically this is published 170 00:10:20,720 --> 00:10:23,600 in 2014, early 2014. 171 00:10:23,600 --> 00:10:32,240 And they found that basically whenever men are working with rodents during psychological 172 00:10:32,240 --> 00:10:36,000 studies, their stress levels are really high. 173 00:10:36,000 --> 00:10:41,120 And we only found this out after more women joined the field and we're working in the 174 00:10:41,120 --> 00:10:42,960 research side of things. 175 00:10:42,960 --> 00:10:48,760 And an old woman research team was trying to recreate some of the studies that have been 176 00:10:48,760 --> 00:10:54,920 done in the 60s and 70s and found that they were having far lower stress levels and they 177 00:10:54,920 --> 00:11:00,840 were releasing different chemicals while those studies were being done. 178 00:11:00,840 --> 00:11:04,640 And so we found that there is basically an entirely different reaction and in some cases 179 00:11:04,640 --> 00:11:11,200 completely different results to entire experiments based around who was involved in the study. 180 00:11:11,200 --> 00:11:16,400 Now how this ties in for me is that up until that study came out and up until that was 181 00:11:16,400 --> 00:11:23,240 found, the entire belief was that rodents were kind of stressed out the whole time and 182 00:11:23,240 --> 00:11:28,320 we were just accepting that as part of what happened. 183 00:11:28,320 --> 00:11:29,760 And it turns out that it wasn't. 184 00:11:29,760 --> 00:11:34,720 So the deep learning that went into all of the, to follow those analogies, the way we 185 00:11:34,720 --> 00:11:39,840 were calculating the equation in our own deep learning and how we were doing our studies 186 00:11:39,840 --> 00:11:44,680 was wrong or at least biased, heavily biased. 187 00:11:44,680 --> 00:11:49,160 And the same type of thing can and probably will and probably already is happening with 188 00:11:49,160 --> 00:11:51,800 the deep learning that's in use. 189 00:11:51,800 --> 00:11:56,160 So we want to take all of this with that in mind, keep in mind that there is a lot of 190 00:11:56,160 --> 00:12:03,960 things we need to be aware of and try not to trust any of this as gospel, I suppose, 191 00:12:03,960 --> 00:12:06,560 for lack of a better word. 192 00:12:06,560 --> 00:12:10,800 We are still letting the computers make decisions based on information we're giving them and 193 00:12:10,800 --> 00:12:13,240 we don't have all the information yet. 194 00:12:13,240 --> 00:12:16,760 I wanted to expand on that a little bit too because I think there's a lot of different 195 00:12:16,760 --> 00:12:17,960 biases that go into it. 196 00:12:17,960 --> 00:12:21,880 And I think Matthew's point at the very, very beginning is quite frankly, we don't know 197 00:12:21,880 --> 00:12:22,880 what we don't know. 198 00:12:22,880 --> 00:12:25,920 And so as we're trying our experiments, we're kind of learning as we go. 199 00:12:25,920 --> 00:12:30,960 I mean, there was an example very similar to the one that Matthew gave where I believe 200 00:12:30,960 --> 00:12:36,280 Amazon created an AI program to help with their interviewing process and they didn't 201 00:12:36,280 --> 00:12:39,960 get too terribly far into it before they realized that there was a heavy bias towards 202 00:12:39,960 --> 00:12:43,640 males just due to the way that they were asking questions. 203 00:12:43,640 --> 00:12:48,640 And it reminded me of another one where sometimes people do this in their normal communications, 204 00:12:48,640 --> 00:12:52,000 you may say something like you guys where you think it's all encompassing, but that's 205 00:12:52,000 --> 00:12:58,040 actually you excluded half the people that are on this podcast just by saying you guys 206 00:12:58,040 --> 00:13:01,280 when the intent was supposed to be everybody, you all, right? 207 00:13:01,280 --> 00:13:02,280 It should have been you all. 208 00:13:02,280 --> 00:13:07,160 And it just happens across the board and it's not necessarily intended to be that way. 209 00:13:07,160 --> 00:13:08,280 Maybe it was, maybe it wasn't. 210 00:13:08,280 --> 00:13:12,440 We grew up with a lot of those kinds of biases in our lives, but it is something that's becoming 211 00:13:12,440 --> 00:13:13,920 very, very prevalent. 212 00:13:13,920 --> 00:13:18,880 Another one that's really interesting to me as well is how you can get some of that just 213 00:13:18,880 --> 00:13:19,920 based on location. 214 00:13:19,920 --> 00:13:25,920 So for example, if we ran an experiment in Alaska, Matthew may come up with some great 215 00:13:25,920 --> 00:13:29,680 examples of how things are going, but it may not apply to somebody in Kentucky. 216 00:13:29,680 --> 00:13:33,640 They may be in a totally different world just because of their locations. 217 00:13:33,640 --> 00:13:39,280 And so again, not really fully understanding how we're gathering that data and how it may 218 00:13:39,280 --> 00:13:43,960 impact something three, four steps down the line will drastically impact the results of 219 00:13:43,960 --> 00:13:44,960 these types of tools. 220 00:13:44,960 --> 00:13:48,600 So I think when we were talking about this at the beginning, as do we rely on this too 221 00:13:48,600 --> 00:13:53,560 much, maybe, although I think it's still a relatively early, but just to be aware of 222 00:13:53,560 --> 00:13:57,240 the context of what it looks like when those kinds of things are happening. 223 00:13:57,240 --> 00:14:02,160 And again, I don't think that the intent originally was designed to be bad or negative. 224 00:14:02,160 --> 00:14:05,800 It just happened because we're humans and we're fallible. 225 00:14:05,800 --> 00:14:11,120 Yeah, Matthew, you had mentioned the 2014 study on the mice. 226 00:14:11,120 --> 00:14:16,960 Todd, you carried on about the inherent human biases even today, right? 227 00:14:16,960 --> 00:14:22,320 With the communication that we have, there's actually, and then kind of bringing tech back 228 00:14:22,320 --> 00:14:23,320 into it. 229 00:14:23,320 --> 00:14:30,880 I wanted to mention, back in 2019, there was a report that NIST, the National Institute 230 00:14:30,880 --> 00:14:36,640 of Standards and Technology, which is a US government agency, they put out a report because 231 00:14:36,640 --> 00:14:44,840 these organizations were trying to sell facial recognition software to the federal government. 232 00:14:44,840 --> 00:14:53,440 And what MIT and NIST ended up doing was they analyzed all this data and said, there are 233 00:14:53,440 --> 00:15:01,040 inherent biases based off of this facial recognition software because it's doing a better job recognizing 234 00:15:01,040 --> 00:15:09,120 white males compared to people in other minority classes, children, elderly, people of color, 235 00:15:09,120 --> 00:15:11,920 women, anything like that. 236 00:15:11,920 --> 00:15:18,080 And so I don't know if women would be included because you guys make up 50% of the population 237 00:15:18,080 --> 00:15:19,080 too. 238 00:15:19,080 --> 00:15:25,600 But anyways, there were inherent biases in the data set because that was the training 239 00:15:25,600 --> 00:15:29,400 model that was typically used to develop the software. 240 00:15:29,400 --> 00:15:31,800 And we've seen that perpetually a little bit. 241 00:15:31,800 --> 00:15:39,080 However, at least we are seeing that the biases are becoming a front and center consideration 242 00:15:39,080 --> 00:15:41,880 as they start to develop new software. 243 00:15:41,880 --> 00:15:48,160 It's not perfect yet, but Matthew, yours started all the way back in the 60s and 70s with just 244 00:15:48,160 --> 00:15:50,040 the raw science. 245 00:15:50,040 --> 00:15:58,520 And now it's taken all this time and it's really becoming apparent that bias in our development 246 00:15:58,520 --> 00:16:00,600 has to be a core consideration. 247 00:16:00,600 --> 00:16:01,600 Yeah. 248 00:16:01,600 --> 00:16:02,600 Yeah. 249 00:16:02,600 --> 00:16:06,440 One of the things I'll throw on there in addition to is it is a real problem, right? 250 00:16:06,440 --> 00:16:10,920 I mean, I think that that problem is going to get worse before it gets better. 251 00:16:10,920 --> 00:16:12,720 And as we touched on, there's all kinds of reasons. 252 00:16:12,720 --> 00:16:15,560 It may be a non-diverse group that's pulling the information together. 253 00:16:15,560 --> 00:16:16,560 You name it. 254 00:16:16,560 --> 00:16:18,120 There's all kinds of issues with it. 255 00:16:18,120 --> 00:16:22,400 But I will say, and we can share this link out too, is there's actually already an open 256 00:16:22,400 --> 00:16:26,080 source tool set out there that's called the AI Fairness tool. 257 00:16:26,080 --> 00:16:30,240 I think it's open AI Fairness 360 or something like that. 258 00:16:30,240 --> 00:16:31,240 We can share the link. 259 00:16:31,240 --> 00:16:35,440 But it's designed to kind of try and go back in time and try to peel back some of those 260 00:16:35,440 --> 00:16:38,280 biases from the tool sets. 261 00:16:38,280 --> 00:16:42,160 When I was kind of getting into the geographical stuff, we kind of at the beginning kind of 262 00:16:42,160 --> 00:16:46,400 talked about tools and how they impact what we're doing today. 263 00:16:46,400 --> 00:16:50,240 And in cybersecurity, there's a lot of AI coming, whether that's deep learning, machine 264 00:16:50,240 --> 00:16:52,720 learning combination of all three is coming. 265 00:16:52,720 --> 00:16:56,800 Matthew mentioned it right out the outchute that you're seeing it in EDR. 266 00:16:56,800 --> 00:17:02,280 There are some very high-end tools that are out there that say very heavily, we are AI 267 00:17:02,280 --> 00:17:03,280 based, right? 268 00:17:03,280 --> 00:17:06,080 And you can see that it's not too terribly far into the future. 269 00:17:06,080 --> 00:17:10,880 It's happening already that you're going to see AI attacking and AI defending. 270 00:17:10,880 --> 00:17:17,160 And when that's the case, you could have some of those biases, concerns like, is it a nation 271 00:17:17,160 --> 00:17:18,160 state? 272 00:17:18,160 --> 00:17:19,720 Someone from Korea is attacking in this way. 273 00:17:19,720 --> 00:17:21,840 Therefore, it might automatically be them. 274 00:17:21,840 --> 00:17:22,840 Absolutely. 275 00:17:22,840 --> 00:17:26,680 And you may get false positives from kind of that type of stuff that's going on. 276 00:17:26,680 --> 00:17:32,760 So from a cybersecurity perspective, the biases are important, maybe not the most important. 277 00:17:32,760 --> 00:17:38,240 I think my takeaway from that, and I might be going a little far, a field at the moment, 278 00:17:38,240 --> 00:17:42,920 but one of my takeaways of that is while these tools are great, you can't use the tool and 279 00:17:42,920 --> 00:17:43,920 depend on it wholly. 280 00:17:43,920 --> 00:17:46,480 You're going to need that human being on the back end. 281 00:17:46,480 --> 00:17:47,480 Exactly. 282 00:17:47,480 --> 00:17:50,560 Go ahead, Matthew. 283 00:17:50,560 --> 00:17:56,360 There's a very specific language you'll find if you are researching EDR software. 284 00:17:56,360 --> 00:17:58,600 And a lot of them will say machine learning. 285 00:17:58,600 --> 00:18:05,560 They will not say deep learning, but they'll also say supervised or unsupervised machine 286 00:18:05,560 --> 00:18:06,760 learning. 287 00:18:06,760 --> 00:18:12,360 And part of that is when we are tracking threats, when we are tracking behavior, there are so 288 00:18:12,360 --> 00:18:20,480 many things that we as humans are, from a computer perspective, inherently chaotic. 289 00:18:20,480 --> 00:18:25,880 We may do the same thing 100,000 times and then we'll randomly go, you know, maybe that 290 00:18:25,880 --> 00:18:28,600 wasn't the best way to do it and just start doing it differently. 291 00:18:28,600 --> 00:18:34,000 And from the computer's perspective, that doesn't make any sense. 292 00:18:34,000 --> 00:18:36,600 So the machine learning side of things, they track that. 293 00:18:36,600 --> 00:18:40,800 They say, actually, this is just someone doing something new for reasons that we don't know 294 00:18:40,800 --> 00:18:42,520 about. 295 00:18:42,520 --> 00:18:46,560 We spend a lot of time on those biases, but that's because that's what ties into all of 296 00:18:46,560 --> 00:18:47,560 this. 297 00:18:47,560 --> 00:18:48,560 We're feeding it data. 298 00:18:48,560 --> 00:18:50,400 We're feeding it language. 299 00:18:50,400 --> 00:18:56,120 And the people there helping with that supervised machine learning are doing the same thing. 300 00:18:56,120 --> 00:18:57,120 They're clicking these buttons. 301 00:18:57,120 --> 00:18:58,320 They're saying, yeah, that's normal. 302 00:18:58,320 --> 00:18:59,320 That's not normal. 303 00:18:59,320 --> 00:19:01,720 Let's try and find this. 304 00:19:01,720 --> 00:19:07,960 Where deep learning really kicks in is when computers are acting autonomously. 305 00:19:07,960 --> 00:19:13,920 Kind of like what Nate and Todd mentioned there, which is if it's doing its own research 306 00:19:13,920 --> 00:19:20,240 and it's trying to make its own actions, deep learning is the way to track a machine doing 307 00:19:20,240 --> 00:19:25,200 something strangely rather than a person doing something strangely. 308 00:19:25,200 --> 00:19:32,800 We're creating systems where they're having to check on themselves, which has its own 309 00:19:32,800 --> 00:19:33,800 problems. 310 00:19:33,800 --> 00:19:39,520 We're just creating Skynet at this point, aren't we? 311 00:19:39,520 --> 00:19:44,200 Those types of items and how we look at it, deep learning is a specific thing where it's 312 00:19:44,200 --> 00:19:46,200 running itself and it's checking for itself. 313 00:19:46,200 --> 00:19:50,960 It's learning on its own items from data that are being fed to it. 314 00:19:50,960 --> 00:19:55,760 And as these become more and more integrated into tools that we use, whether it's network 315 00:19:55,760 --> 00:20:01,640 scanning, whether it's anything else, we're going to start seeing times when a false positive 316 00:20:01,640 --> 00:20:04,120 maybe knocks someone out of a system. 317 00:20:04,120 --> 00:20:10,240 Maybe the CEO is traveling and signs into the wrong spot and so the system fully locks 318 00:20:10,240 --> 00:20:13,440 them out. 319 00:20:13,440 --> 00:20:15,720 There are things to keep in mind for that as well. 320 00:20:15,720 --> 00:20:23,920 Are you and is your system and your SOC team prepared and ready to handle if a deep learning 321 00:20:23,920 --> 00:20:28,320 change resulted in the entire system deciding something was bad that had previously been 322 00:20:28,320 --> 00:20:29,320 good? 323 00:20:29,320 --> 00:20:30,320 Yeah. 324 00:20:30,320 --> 00:20:42,000 So, Matthew, you and Todd have both mentioned false positives already. 325 00:20:42,000 --> 00:20:48,800 And while you guys are talking, and I promise marketing team that I'll give you a chance 326 00:20:48,800 --> 00:20:51,280 to ask questions here shortly. 327 00:20:51,280 --> 00:20:56,680 I tend to be a glass half full kind of person most of the time, right? 328 00:20:56,680 --> 00:21:02,760 I embrace the AI, the machine learning, the deep learning, all that kind of stuff. 329 00:21:02,760 --> 00:21:05,120 So do I think we trust it too much? 330 00:21:05,120 --> 00:21:14,880 I'll summarize my opinion very early into the podcast is yes, I do, but I still want 331 00:21:14,880 --> 00:21:18,720 to embrace it because the technology is phenomenal, right? 332 00:21:18,720 --> 00:21:25,080 And so, but there are careful considerations, which we've been talking about, but kind of 333 00:21:25,080 --> 00:21:29,520 getting back to the false positive versus that you guys have both mentioned. 334 00:21:29,520 --> 00:21:31,680 I also want to mention false negative. 335 00:21:31,680 --> 00:21:38,040 So for those that aren't familiar with it, we have false positive is something has flagged 336 00:21:38,040 --> 00:21:43,360 and alert, but it was falsely correlated, right? 337 00:21:43,360 --> 00:21:50,120 Maybe it wasn't something malicious that it thought wasn't malicious. 338 00:21:50,120 --> 00:21:53,000 The more dangerous one is false negative, right? 339 00:21:53,000 --> 00:22:00,960 Is saying that I thought something was legitimate or non malicious and it led it through. 340 00:22:00,960 --> 00:22:07,320 And so, when we start talking about do we trust deep learning or AI or machine learning 341 00:22:07,320 --> 00:22:09,360 too much? 342 00:22:09,360 --> 00:22:14,160 Those are the considerations that I do want to have at least people aware of or, you know, 343 00:22:14,160 --> 00:22:19,080 vendors that are developing this software being aware of saying as the tools are trying 344 00:22:19,080 --> 00:22:26,000 to learn this stuff, we have seen cases where false positives are common, right? 345 00:22:26,000 --> 00:22:28,840 Everyone wants to try and be as restrictive as possible. 346 00:22:28,840 --> 00:22:30,800 So those are very, very frequent, right? 347 00:22:30,800 --> 00:22:36,720 So maybe there's two anomalous activities that are miscorrelated that look malicious 348 00:22:36,720 --> 00:22:40,080 together, but they are two independent events. 349 00:22:40,080 --> 00:22:44,680 But the more dangerous one is that there's two independent malicious things happening 350 00:22:44,680 --> 00:22:50,560 or maybe just one that slips under the radar that doesn't quite seem malicious, but maybe 351 00:22:50,560 --> 00:22:55,160 it's a precursor to a larger attack. 352 00:22:55,160 --> 00:23:05,960 And so that's where I guess I'm still glass half full, but we have seen cases even here 353 00:23:05,960 --> 00:23:12,520 at CIT as we're monitoring customer networks that both could come into play. 354 00:23:12,520 --> 00:23:17,160 But that's where Todd, you had mentioned the human element still has to come into play 355 00:23:17,160 --> 00:23:23,280 to give that deeper review to confirm the potential inherent bias that the human put 356 00:23:23,280 --> 00:23:25,320 into the first place. 357 00:23:25,320 --> 00:23:26,800 Yeah, I agree. 358 00:23:26,800 --> 00:23:29,720 I mean, for what it's worth, and I think we've talked about this in previous podcasts, 359 00:23:29,720 --> 00:23:32,280 too, is I think we're all tech nerds and we love the technology. 360 00:23:32,280 --> 00:23:34,120 And I think it's a game changer, right? 361 00:23:34,120 --> 00:23:38,840 I mean, it absolutely has massive, massive upside. 362 00:23:38,840 --> 00:23:40,480 Am I concerned about SkyNet? 363 00:23:40,480 --> 00:23:41,480 100%. 364 00:23:41,480 --> 00:23:44,360 Am I concerned it's happening today or tomorrow? 365 00:23:44,360 --> 00:23:46,400 No, absolutely not. 366 00:23:46,400 --> 00:23:51,120 I do tend to fall into the NAIC category, too, is I think there's a lot more good than 367 00:23:51,120 --> 00:23:52,120 bad. 368 00:23:52,120 --> 00:23:56,040 I think that when I'm bringing up the cautionary things that are coming up in it is, in my 369 00:23:56,040 --> 00:24:00,200 opinion is, it means that there's a reason to pay attention. 370 00:24:00,200 --> 00:24:01,640 There are concerns there. 371 00:24:01,640 --> 00:24:03,920 You still need, you can't just flip it on and go, oh, we're all good. 372 00:24:03,920 --> 00:24:06,980 We've got this magical computer taking care of us now. 373 00:24:06,980 --> 00:24:08,320 There are things that you need to watch. 374 00:24:08,320 --> 00:24:13,000 You need to continue to see how it's evolving, what's changing in the industry, et cetera, 375 00:24:13,000 --> 00:24:14,000 et cetera. 376 00:24:14,000 --> 00:24:18,720 So from that perspective, I think that awareness helps you with the risk categorization, if 377 00:24:18,720 --> 00:24:19,800 that makes sense. 378 00:24:19,800 --> 00:24:21,520 There are risks associated with it. 379 00:24:21,520 --> 00:24:31,440 It's doing a nice job, but it is not infallible, just like the data that we fed it. 380 00:24:31,440 --> 00:24:32,440 I promise marketing. 381 00:24:32,440 --> 00:24:36,680 You promised and then I was like, dang, good questions ready. 382 00:24:36,680 --> 00:24:38,680 Todd ruined that one. 383 00:24:38,680 --> 00:24:39,960 Nope, Todd took over. 384 00:24:39,960 --> 00:24:41,200 I promise to you. 385 00:24:41,200 --> 00:24:42,800 Yeah, that's okay. 386 00:24:42,800 --> 00:24:47,720 We let Todd talk, but I did have kind of a tangent question of, okay, so we've mentioned 387 00:24:47,720 --> 00:24:52,200 EDR and the sake of that, right, CIT is a tech provider. 388 00:24:52,200 --> 00:24:53,320 We work with other businesses. 389 00:24:53,320 --> 00:24:54,400 We install these tools. 390 00:24:54,400 --> 00:24:56,000 We monitor it. 391 00:24:56,000 --> 00:25:01,080 In that case, how would you know if your tech provider is trusting deep learning too much 392 00:25:01,080 --> 00:25:06,240 when you're not the person necessarily being that first line looking at the tool? 393 00:25:06,240 --> 00:25:07,240 Ask. 394 00:25:07,240 --> 00:25:08,240 Thank you. 395 00:25:08,240 --> 00:25:15,480 I mean, I'm more than happy to answer those questions when they come through. 396 00:25:15,480 --> 00:25:19,520 I mean, we do get them sometimes, we do have, you know, there's due diligence questions 397 00:25:19,520 --> 00:25:22,560 that come through and a lot of the compliance stuff I'm looking at. 398 00:25:22,560 --> 00:25:27,080 And so not afraid of it, right? 399 00:25:27,080 --> 00:25:32,600 Any tech company, in my opinion, that's worth their salt isn't going to get upset or push 400 00:25:32,600 --> 00:25:36,160 back when you ask questions about their process. 401 00:25:36,160 --> 00:25:43,800 On top of that, fully support full vulnerability scans or, you know, penetration testing that 402 00:25:43,800 --> 00:25:47,600 can be done by third parties to find out where things are going on. 403 00:25:47,600 --> 00:25:54,080 And this is just part of that due diligence of we are doing our due diligence. 404 00:25:54,080 --> 00:25:55,080 We're looking into it. 405 00:25:55,080 --> 00:25:56,080 We're testing it. 406 00:25:56,080 --> 00:25:58,920 We're finding that level that works for us. 407 00:25:58,920 --> 00:26:02,160 So if you ask us about it, we'll have an answer. 408 00:26:02,160 --> 00:26:06,440 My first step would be to ask them if they have a podcast called, do you trust deep learning 409 00:26:06,440 --> 00:26:09,880 too much? 410 00:26:09,880 --> 00:26:13,440 So yeah, I would, I would kind of add on to that. 411 00:26:13,440 --> 00:26:16,680 I would say that you really should ask. 412 00:26:16,680 --> 00:26:20,000 I think it's not as simple as saying, do you or don't you? 413 00:26:20,000 --> 00:26:23,720 Because a lot of the tech companies out there are actually outsourcing a lot of their security. 414 00:26:23,720 --> 00:26:27,320 And so they may not have a security team on board. 415 00:26:27,320 --> 00:26:29,800 They may just send it to somebody else. 416 00:26:29,800 --> 00:26:33,120 And so depending on how that question goes is who's handling that for you? 417 00:26:33,120 --> 00:26:35,840 Is it is it people on your team? 418 00:26:35,840 --> 00:26:38,280 To me that it does have some importance now. 419 00:26:38,280 --> 00:26:39,360 We do have it on our team. 420 00:26:39,360 --> 00:26:43,320 So that that probably again, going into the bias that probably biases me. 421 00:26:43,320 --> 00:26:48,920 Specifically, but I feel like our customers turn to us very specifically and say, you 422 00:26:48,920 --> 00:26:51,800 tell me, I this is not my field of expertise. 423 00:26:51,800 --> 00:26:54,400 I need you to tell me like I'd go to a lawyer, right? 424 00:26:54,400 --> 00:26:56,000 I'm going to go, Hey, you're my lawyer. 425 00:26:56,000 --> 00:26:59,440 You can't just lob that back to me and say, I'm not going to answer your question or I'll 426 00:26:59,440 --> 00:27:00,880 answer it with another question. 427 00:27:00,880 --> 00:27:03,160 I'm coming to you because you're my expert. 428 00:27:03,160 --> 00:27:05,320 And I think that's what our customers are doing. 429 00:27:05,320 --> 00:27:10,320 So having that on staff, IT person or security individuals incredibly important. 430 00:27:10,320 --> 00:27:13,760 So that's how I would find out the answer to that question. 431 00:27:13,760 --> 00:27:14,760 Yeah. 432 00:27:14,760 --> 00:27:17,760 And, you know, I had mentioned that I'm a glass half full. 433 00:27:17,760 --> 00:27:20,600 I embrace the technology. 434 00:27:20,600 --> 00:27:24,080 I am still very hesitant of it. 435 00:27:24,080 --> 00:27:30,640 We have tools that are deep into the machine learning here at CIT to try and better detect 436 00:27:30,640 --> 00:27:33,040 threats within our environment. 437 00:27:33,040 --> 00:27:41,600 We've got layers on layers on layers of tech to try and identify security threats here. 438 00:27:41,600 --> 00:27:48,640 Even then, I don't trust the vendors that start claiming AI, you know, they're going 439 00:27:48,640 --> 00:27:54,000 to solve all the issues where one stop shop. 440 00:27:54,000 --> 00:27:57,320 Security is all about the multi-layered approach. 441 00:27:57,320 --> 00:28:02,640 And it's one of the things where even though I do trust it, so sorry, I don't trust it. 442 00:28:02,640 --> 00:28:09,480 I just said that earlier, even though I embrace it so much, it's still one of the things where 443 00:28:09,480 --> 00:28:17,040 if someone claims that their tool can stop either 100% of all threats out there, I don't 444 00:28:17,040 --> 00:28:18,040 trust it. 445 00:28:18,040 --> 00:28:24,440 And as your technology provider trying to educate, come ask us and we'll tell you, you know, 446 00:28:24,440 --> 00:28:30,200 where are the potential pitfalls of a solution and where should you fill in with another 447 00:28:30,200 --> 00:28:34,640 solution to accommodate that next threat, right? 448 00:28:34,640 --> 00:28:35,640 Yeah. 449 00:28:35,640 --> 00:28:38,200 So, it's all about the layers. 450 00:28:38,200 --> 00:28:39,200 Yeah. 451 00:28:39,200 --> 00:28:40,200 I like the analogy. 452 00:28:40,200 --> 00:28:44,680 You can't put the genie back in the bottle just because you don't want the wishes, right? 453 00:28:44,680 --> 00:28:51,040 These products are out there and they have a lot of the ones that, you know, OpenAI has 454 00:28:51,040 --> 00:28:54,480 a very easy to use interface. 455 00:28:54,480 --> 00:28:55,480 People are going to use it. 456 00:28:55,480 --> 00:28:56,960 You can't put it away. 457 00:28:56,960 --> 00:29:02,040 So let's find ways to figure out how to introduce it safely. 458 00:29:02,040 --> 00:29:06,240 Come up with ways that you can have your organization either implement or specifically 459 00:29:06,240 --> 00:29:11,480 say do not implement parts of what it does. 460 00:29:11,480 --> 00:29:17,120 But it's out there and so we need to be aware and we need to be thinking about it. 461 00:29:17,120 --> 00:29:18,120 It's going to keep happening. 462 00:29:18,120 --> 00:29:21,080 There's going to be things that come out and there are things that come out like this for 463 00:29:21,080 --> 00:29:28,920 us regularly staying on top of these things is literally the job. 464 00:29:28,920 --> 00:29:34,920 As fun as it is sometimes to use these tools, there are things that I use for home that 465 00:29:34,920 --> 00:29:39,800 I would never use for work and making that distinction and having those conversations 466 00:29:39,800 --> 00:29:43,440 is a critical part of doing your due diligence. 467 00:29:43,440 --> 00:29:44,440 Yeah. 468 00:29:44,440 --> 00:29:50,280 I think part of doing your own due diligence as well as run the tool through its paces. 469 00:29:50,280 --> 00:29:54,800 If you get a proof of concept on something, try and break it. 470 00:29:54,800 --> 00:30:00,400 Try and figure out how you can bypass it. 471 00:30:00,400 --> 00:30:05,160 We've taken a look at plenty of security solutions over the years and whenever I get a new one, 472 00:30:05,160 --> 00:30:08,960 I say here's some common attack paths. 473 00:30:08,960 --> 00:30:11,200 By the way, your tool didn't account for that. 474 00:30:11,200 --> 00:30:12,480 Your tool didn't account for that. 475 00:30:12,480 --> 00:30:14,560 Your tool didn't account for that. 476 00:30:14,560 --> 00:30:20,520 Then eventually we find one that's, wow, this is a very phenomenal tool. 477 00:30:20,520 --> 00:30:25,400 Otherwise, if you're talking about EDR, that's just one topic. 478 00:30:25,400 --> 00:30:27,960 There's plenty of security solutions out there. 479 00:30:27,960 --> 00:30:33,920 Taking a look at unbiased third-party testing solutions. 480 00:30:33,920 --> 00:30:37,120 We've talked EDR many, many times in the past. 481 00:30:37,120 --> 00:30:41,320 Go take a look at the MITRE Injunuity test results. 482 00:30:41,320 --> 00:30:42,320 It's unbiased. 483 00:30:42,320 --> 00:30:45,240 They just say, here's the results. 484 00:30:45,240 --> 00:30:47,560 Vendors submit their tool. 485 00:30:47,560 --> 00:30:50,160 There's no opinions about it. 486 00:30:50,160 --> 00:30:55,240 Then from there, you can formulate your own opinions. 487 00:30:55,240 --> 00:30:58,200 Take third-party research into consideration as well. 488 00:30:58,200 --> 00:31:01,640 Don't just trust the vendor. 489 00:31:01,640 --> 00:31:09,920 As a final caveat on all this, I want to add, remember Richard Sarah in the early 70s was 490 00:31:09,920 --> 00:31:15,360 talking about television and said, if something is free, you're the product. 491 00:31:15,360 --> 00:31:16,920 Please remember that. 492 00:31:16,920 --> 00:31:21,280 Don't feed your business critical information into a product that they're giving you for 493 00:31:21,280 --> 00:31:25,320 free because there's a good chance they're using that data for something else. 494 00:31:25,320 --> 00:31:31,640 This is a nice little paranoid warning to end my part of the podcast. 495 00:31:31,640 --> 00:31:33,640 Thank you and good night. 496 00:31:33,640 --> 00:31:34,640 Ah. 497 00:31:34,640 --> 00:31:35,640 I love it. 498 00:31:35,640 --> 00:31:39,080 I love it. 499 00:31:39,080 --> 00:31:43,320 It feels like a great little closer, important to keep in mind. 500 00:31:43,320 --> 00:31:47,960 I want to just echo communication is key. 501 00:31:47,960 --> 00:31:53,040 Asking, and if you have a question or a concern, whether you're a customer or not, please reach 502 00:31:53,040 --> 00:32:00,720 out to us at info at cit-net.com or you can head out to our website, cit-net.com, slash 503 00:32:00,720 --> 00:32:01,720 podcast. 504 00:32:01,720 --> 00:32:05,280 Thank you, Todd, Nate, and Matthew for joining us today. 505 00:32:05,280 --> 00:32:12,960 We'll be back next week with an all-new episode.