1 00:00:00,000 --> 00:00:09,400 Welcome to the Azure Security Podcast where we discuss topics relating to security, privacy, 2 00:00:09,400 --> 00:00:13,640 reliability, and compliance on the Microsoft Cloud Platform. 3 00:00:13,640 --> 00:00:17,080 Hey everybody, welcome to episode 29. 4 00:00:17,080 --> 00:00:18,080 We have a full house this week. 5 00:00:18,080 --> 00:00:20,880 We have myself, Sarah Mark and Gladys. 6 00:00:20,880 --> 00:00:26,720 We also have a guest, Minnie Wallyer, and she's here to talk to us about Azure Data Explorer. 7 00:00:26,720 --> 00:00:29,920 Before we get to Minnie, let's take a moment and go through the news. 8 00:00:29,920 --> 00:00:32,240 Mike, why don't you kick us off? 9 00:00:32,240 --> 00:00:37,120 First thing is the long awaited and took a lot of work. 10 00:00:37,120 --> 00:00:40,800 Cyber Reference Architecture, the Microsoft Cyber Reference Architecture, affectionately 11 00:00:40,800 --> 00:00:43,160 known as MCRA, is out. 12 00:00:43,160 --> 00:00:44,160 It's released. 13 00:00:44,160 --> 00:00:47,360 So AKMS slash MCRA is out. 14 00:00:47,360 --> 00:00:51,920 So it's got the original capability one that everybody's familiar with, or many people 15 00:00:51,920 --> 00:00:52,920 are. 16 00:00:52,920 --> 00:00:56,160 We had like 80,000 downloads in the last version, so plenty. 17 00:00:56,160 --> 00:01:01,960 But we also added pretty much all of the sort of dense complex, bring it all together diagram. 18 00:01:01,960 --> 00:01:06,440 So Azure Security, native controls, like what are the stuff that's built into Azure and 19 00:01:06,440 --> 00:01:10,240 to our cloud that you can use to protect your Azure end-to-end, everything from the user 20 00:01:10,240 --> 00:01:16,120 accounts to the devices, to the backend resources, to the apps, to the IoT and OT devices that 21 00:01:16,120 --> 00:01:17,120 connect to it. 22 00:01:17,120 --> 00:01:18,520 So we've got that. 23 00:01:18,520 --> 00:01:24,440 We've got security operations or SOC reference architecture, zero trust user access. 24 00:01:24,440 --> 00:01:30,920 We've got some kill chain stuff in there, a people diagram, so how the roles and responsibilities 25 00:01:30,920 --> 00:01:37,000 fit within an organization and work together and what are the jobs to be done. 26 00:01:37,000 --> 00:01:41,120 And a whole lot of zero trust stuff and some other security operations and threat intelligence. 27 00:01:41,120 --> 00:01:46,640 So pretty much kind of a best of cornucopia of technical goodness. 28 00:01:46,640 --> 00:01:47,880 And so that one is, it's out there. 29 00:01:47,880 --> 00:01:49,080 It's available, ready for download. 30 00:01:49,080 --> 00:01:50,880 I'd love to get any feedback on it. 31 00:01:50,880 --> 00:01:54,120 So hit me up on the socials or whatever for that. 32 00:01:54,120 --> 00:02:00,000 Second one is, because the reference architecture is a nice kind of architectural level view, 33 00:02:00,000 --> 00:02:03,880 kind of middle of the security org from a top bottom perspective. 34 00:02:03,880 --> 00:02:09,280 And then we realize that a lot of organizations are kind of looking for guidance sort of from 35 00:02:09,280 --> 00:02:13,840 a top down perspective and how do you organize your program and set up goals and metrics 36 00:02:13,840 --> 00:02:16,680 and budgets and all that kind of good stuff. 37 00:02:16,680 --> 00:02:19,680 And also how to interact with the business. 38 00:02:19,680 --> 00:02:24,360 And so what we did in our cloud adoption framework was we added a secure methodology. 39 00:02:24,360 --> 00:02:31,160 And so just like we have strategy plan, build, manage, organize, et cetera, we now have secure. 40 00:02:31,160 --> 00:02:35,440 So as organizations go to the cloud, this is now a native component of it. 41 00:02:35,440 --> 00:02:38,440 This is how to do the security part. 42 00:02:38,440 --> 00:02:42,440 And so we cover a lot of stuff in there. 43 00:02:42,440 --> 00:02:46,080 There's kind of like a top half of it, which is the risk insights and how do you align 44 00:02:46,080 --> 00:02:51,280 to the business and the initiatives and the priorities and the risk registers and language 45 00:02:51,280 --> 00:02:53,280 and prioritization of risk. 46 00:02:53,280 --> 00:02:56,920 And then security integration, kind of how do you do that deeper in the org, business 47 00:02:56,920 --> 00:02:58,640 resilience and how to think about that. 48 00:02:58,640 --> 00:03:02,240 So these are the things that essentially security provides to the business and what the business 49 00:03:02,240 --> 00:03:06,120 should expect of sort of the CISO and team. 50 00:03:06,120 --> 00:03:11,800 And then we also have kind of the bottom half of the lower half that's focused on the security 51 00:03:11,800 --> 00:03:12,960 disciplines. 52 00:03:12,960 --> 00:03:21,400 And we aligned it to NIST, but a little bit more closely actually to the open group where 53 00:03:21,400 --> 00:03:26,440 we wanted to have like really actionable specific disciplines that are both familiar but also 54 00:03:26,440 --> 00:03:29,000 push the organization into the future. 55 00:03:29,000 --> 00:03:31,520 And so access control is the first one. 56 00:03:31,520 --> 00:03:34,720 And that's really where networking and identity really need to come together. 57 00:03:34,720 --> 00:03:39,120 Not that sometimes they're on the same page, but oftentimes not in an organization, but 58 00:03:39,120 --> 00:03:42,960 really to kind of have this sort of end to end view of how do we actually provide access 59 00:03:42,960 --> 00:03:46,560 in this age of cloud, security operations, how do you handle the incidents that are 60 00:03:46,560 --> 00:03:51,000 coming in, the active realize risks, asset protection, how do you think about this in 61 00:03:51,000 --> 00:03:55,120 sort of a dynamic environment with things popping up in infrastructure as code all the 62 00:03:55,120 --> 00:03:58,800 time, new services, new SaaS apps. 63 00:03:58,800 --> 00:04:01,320 So how do you really think about that and do it right? 64 00:04:01,320 --> 00:04:02,640 Security governance. 65 00:04:02,640 --> 00:04:04,600 And we kind of made governance kind of fun. 66 00:04:04,600 --> 00:04:08,120 And I know it's a little crazy to say that, but we did kind of help bridge it to the 67 00:04:08,120 --> 00:04:12,240 business, talked about how to deal with like continuous change. 68 00:04:12,240 --> 00:04:15,000 And what are the kind of key hallmarks of success there and kind of how to think about 69 00:04:15,000 --> 00:04:16,480 that program. 70 00:04:16,480 --> 00:04:20,520 And then we also have innovation security, which is really getting into the DevSec op 71 00:04:20,520 --> 00:04:21,520 space. 72 00:04:21,520 --> 00:04:23,680 And, you know, what does good look like there? 73 00:04:23,680 --> 00:04:25,720 And there's some cultural elements to it. 74 00:04:25,720 --> 00:04:29,000 There's some technical elements to it and some process pieces. 75 00:04:29,000 --> 00:04:32,680 So definitely check out the CAF secure stuff. 76 00:04:32,680 --> 00:04:34,320 Links are both convenient to show us. 77 00:04:34,320 --> 00:04:35,320 Okay. 78 00:04:35,320 --> 00:04:37,080 So time for some news from me. 79 00:04:37,080 --> 00:04:41,320 I'm going to go for not my baby just to start with. 80 00:04:41,320 --> 00:04:45,920 By the way, I've now been having some emails from some of the people out there in the big 81 00:04:45,920 --> 00:04:51,120 wide world actually referring to Sentinel as my baby in emails, which I find hugely 82 00:04:51,120 --> 00:04:53,120 amusing and thank you very much. 83 00:04:53,120 --> 00:04:56,720 And I should probably stop referring to it as such though. 84 00:04:56,720 --> 00:05:00,000 Anyway, let's talk about Azure Security Center. 85 00:05:00,000 --> 00:05:05,160 So there's some cool things that have come into preview and GA this month. 86 00:05:05,160 --> 00:05:10,720 So for a start, we've got a new resource health page that's in preview. 87 00:05:10,720 --> 00:05:15,680 So that's not something that we've had before, but now it's much prettier. 88 00:05:15,680 --> 00:05:17,600 It has a nicer view. 89 00:05:17,600 --> 00:05:21,680 So go and check it out if you're using Azure Security Center. 90 00:05:21,680 --> 00:05:26,560 And if you're using Azure Defender, you can also now see on one page the outstanding security 91 00:05:26,560 --> 00:05:30,040 alerts, which is pretty neat. 92 00:05:30,040 --> 00:05:35,720 The other thing that in preview is using Azure Defender for Kubernetes. 93 00:05:35,720 --> 00:05:40,800 Now that's been around for a while, but now you can use it to protect hybrid and multi-cloud 94 00:05:40,800 --> 00:05:45,840 Kubernetes deployments before it was just using AKS. 95 00:05:45,840 --> 00:05:52,800 So if you've got Kubernetes running in another cloud or you have it maybe on-prem, I mean 96 00:05:52,800 --> 00:05:55,440 you could have it on-prem, I guess. 97 00:05:55,440 --> 00:06:00,360 You can now, using Azure Arc, actually have that protected with Azure Defender. 98 00:06:00,360 --> 00:06:03,320 So again, very cool. 99 00:06:03,320 --> 00:06:09,800 We've also got some more recommendations around Azure Defender for DNS and resource manager. 100 00:06:09,800 --> 00:06:17,040 So if you're using Azure DNS, we've now got Azure Defender for DNS and we've got Azure 101 00:06:17,040 --> 00:06:18,720 Defender for resource manager. 102 00:06:18,720 --> 00:06:23,600 So of course, if you're using Azure Defender, go and look at turning it on. 103 00:06:23,600 --> 00:06:26,720 There are charges, so do have a look. 104 00:06:26,720 --> 00:06:29,000 I think it's always important to say that. 105 00:06:29,000 --> 00:06:34,600 Something else that's just cool for my part of the world is that they've also added some 106 00:06:34,600 --> 00:06:38,920 new regulatory compliance standards into Security Center. 107 00:06:38,920 --> 00:06:45,280 So if you're unfamiliar with this feature, it basically allows Security Center will rate 108 00:06:45,280 --> 00:06:48,480 the infrastructure that it has, services it's tracking. 109 00:06:48,480 --> 00:06:51,080 It will rate it against regulatory compliance standards. 110 00:06:51,080 --> 00:06:56,840 So we've had a lot of ones unsurprisingly focused on the US and Europe to start with. 111 00:06:56,840 --> 00:07:01,560 So NIST, we've got HIPAA, et cetera. 112 00:07:01,560 --> 00:07:07,320 But just yay for my part of the world, they've now added the New Zealand ISM, the Information 113 00:07:07,320 --> 00:07:08,880 Security Manual. 114 00:07:08,880 --> 00:07:15,360 So for those of you in New Zealand, you can now go and rate things against that government 115 00:07:15,360 --> 00:07:17,520 standard. 116 00:07:17,520 --> 00:07:23,120 So just because I'm focusing way too much on New Zealand, we've also got the Azure CIS, 117 00:07:23,120 --> 00:07:28,880 the Center for Internet Security Benchmarks, and CMMC level three. 118 00:07:28,880 --> 00:07:33,760 That's probably my interesting things for Security Center. 119 00:07:33,760 --> 00:07:38,200 So moving on to another thing that I'm a big fan of is AKS. 120 00:07:38,200 --> 00:07:45,440 So AKS now has support for the secret store CSI that's gone into public preview. 121 00:07:45,440 --> 00:07:51,280 So that means that if you're using a container storage interface driver, you can mount secrets, 122 00:07:51,280 --> 00:07:57,440 keys, inserts stored in your secret stores into your pod as a CSI volume. 123 00:07:57,440 --> 00:08:03,000 So that just basically means that it's going to make your secure access to secrets much, 124 00:08:03,000 --> 00:08:05,000 much, much more straightforward. 125 00:08:05,000 --> 00:08:09,400 And you can do it via the containers file system. 126 00:08:09,400 --> 00:08:14,600 And then we've got to go on to, of course, my baby, something that one of our colleagues 127 00:08:14,600 --> 00:08:21,920 here in Microsoft has created is the Azure Sentinel Sock Process Framework, and that's 128 00:08:21,920 --> 00:08:23,280 been created by RIN. 129 00:08:23,280 --> 00:08:27,760 Now he's actually going to come, he's going to be a guest on the podcast to talk about 130 00:08:27,760 --> 00:08:33,960 this in lots of detail in a few episodes time, but definitely go and check out what he's 131 00:08:33,960 --> 00:08:34,960 done. 132 00:08:34,960 --> 00:08:41,680 He's basically created this whole Sock Process Framework workbook that will show you what 133 00:08:41,680 --> 00:08:47,440 you're doing, what you're missing, things that you need to add into your Sock to make 134 00:08:47,440 --> 00:08:49,680 it a good, mature process. 135 00:08:49,680 --> 00:08:51,360 And it's really, really interesting stuff. 136 00:08:51,360 --> 00:08:56,680 I know that he's also been doing it with some of the other great people who work on Azure 137 00:08:56,680 --> 00:09:00,240 Sentinel throughout the world. 138 00:09:00,240 --> 00:09:02,640 And so definitely go check out his blog post. 139 00:09:02,640 --> 00:09:05,720 It's now also just because I did it myself. 140 00:09:05,720 --> 00:09:10,000 It's just been added to the official Azure Sentinel repo. 141 00:09:10,000 --> 00:09:14,480 So in the not too distant future, you should actually see the workbook in the Sentinel 142 00:09:14,480 --> 00:09:19,360 UI, but if you want to look at it before then, and I really recommend you do, have 143 00:09:19,360 --> 00:09:24,840 a look at the link to the blog post that we have in the show notes and go and check it 144 00:09:24,840 --> 00:09:27,600 out because it's a very impressive piece of work. 145 00:09:27,600 --> 00:09:34,040 And I'm really looking forward to talking to RIN about it in a few episodes time. 146 00:09:34,040 --> 00:09:36,360 And yeah, that's all of my news. 147 00:09:36,360 --> 00:09:38,360 So over to you, Michael. 148 00:09:38,360 --> 00:09:41,560 A whole bunch of things caught my interest over the last couple of weeks. 149 00:09:41,560 --> 00:09:46,000 The first is an announcement that came out of Microsoft Build this week, which was Azure 150 00:09:46,000 --> 00:09:48,280 SQL Database Ledger. 151 00:09:48,280 --> 00:09:52,320 Probably one of the best ways of thinking about this is imagine if you took a SQL database 152 00:09:52,320 --> 00:09:58,160 and add basically kind of the power of blockchain to a SQL database. 153 00:09:58,160 --> 00:10:03,480 This allows you to have records in the database that you can show have not been changed. 154 00:10:03,480 --> 00:10:07,360 Or if you do make an update, you can show the complete record of the changes that happen 155 00:10:07,360 --> 00:10:08,880 in the database. 156 00:10:08,880 --> 00:10:13,800 This is really fantastic for people for say regulatory requirements where they can show 157 00:10:13,800 --> 00:10:20,320 the chain of custody, show the lineage of data changes within a database. 158 00:10:20,320 --> 00:10:25,880 The next one is new security features in Azure VPN gateway. 159 00:10:25,880 --> 00:10:30,600 So for example, you can now have multiple authentication types on a single gateway for 160 00:10:30,600 --> 00:10:32,640 open VPN tunnels. 161 00:10:32,640 --> 00:10:36,720 So you could have Azure AD, you can have certificate based and radius based authentication 162 00:10:36,720 --> 00:10:38,920 all on the same gateway. 163 00:10:38,920 --> 00:10:43,000 There's also border gateway protocol diagnostic support in there. 164 00:10:43,000 --> 00:10:48,600 And we've also added VPN packet capture and much better VPN connection management as well. 165 00:10:48,600 --> 00:10:52,680 So a lot of good improvements that I know a lot of customers have been asking for. 166 00:10:52,680 --> 00:11:00,200 We've also added a new training class to help people learn using Bicep. 167 00:11:00,200 --> 00:11:03,360 So Bicep is just essentially the next generation of ARM templates. 168 00:11:03,360 --> 00:11:11,320 It's currently in preview and we're taking a lot of suggestions from our customers. 169 00:11:11,320 --> 00:11:15,400 Do be aware that if you're going to use Bicep that there is a really good chance that there 170 00:11:15,400 --> 00:11:17,160 will be breaking changes. 171 00:11:17,160 --> 00:11:23,120 But Bicep is essentially a more modern, more clean way of building templates for deploying 172 00:11:23,120 --> 00:11:25,240 infrastructure as code. 173 00:11:25,240 --> 00:11:28,560 We've also added some enhancements to Azure backup. 174 00:11:28,560 --> 00:11:34,080 Just notably we now have support for managed identities, for managing permissions on keys 175 00:11:34,080 --> 00:11:36,280 that are used in recovery services. 176 00:11:36,280 --> 00:11:40,920 We've also now enabled encryption using customer managed keys during creation of the recovery 177 00:11:40,920 --> 00:11:43,040 services vault. 178 00:11:43,040 --> 00:11:45,800 That is in limited preview. 179 00:11:45,800 --> 00:11:49,720 If you're interested, there's a link in the show notes through the article that has the 180 00:11:49,720 --> 00:11:54,840 email account that you can email to opt in for using or kicking the tires on this particular 181 00:11:54,840 --> 00:11:56,120 feature. 182 00:11:56,120 --> 00:12:02,840 And finally, there's now been added new Azure policy support to enforce encryption backups 183 00:12:02,840 --> 00:12:04,440 using customer managed keys. 184 00:12:04,440 --> 00:12:08,760 We've also added some new security features to API management. 185 00:12:08,760 --> 00:12:13,560 Most notably, there's now the ability to validate a client certificate. 186 00:12:13,560 --> 00:12:18,800 So if you're using TLS, which of course you should be, which will give you server authentication, 187 00:12:18,800 --> 00:12:22,240 and you opt in for, you can now opt in for using a client certificate for authentication 188 00:12:22,240 --> 00:12:23,740 as well. 189 00:12:23,740 --> 00:12:30,520 We've also added a update to the ciphers and protocols page inside of the Azure portal 190 00:12:30,520 --> 00:12:37,840 that makes it much easier for managing cipher suites and protocol versions when using TLS. 191 00:12:37,840 --> 00:12:44,680 We will also warn you about using weak cipher suites or weak protocol versions. 192 00:12:44,680 --> 00:12:51,320 The last one is Azure key vault is now updated at service level agreement. 193 00:12:51,320 --> 00:12:56,520 It was historically 99.9% SLA. 194 00:12:56,520 --> 00:13:04,560 We've now bumped that up to 99.99 SLA, which is fantastic for those people who are using 195 00:13:04,560 --> 00:13:09,600 Azure key vaults in mission critical environments that require a high level of SLA. 196 00:13:09,600 --> 00:13:12,080 So that's all I have for the news. 197 00:13:12,080 --> 00:13:13,840 Glad it's over to you. 198 00:13:13,840 --> 00:13:21,280 I wanted to talk about CMLAM, which is an open source initiative to help security researchers 199 00:13:21,280 --> 00:13:30,000 deploy labs in where they can reproduce well known techniques used in real attack scenarios 200 00:13:30,000 --> 00:13:37,680 in order to test and verify the effectiveness of services like Microsoft 365 Defender, Azure 201 00:13:37,680 --> 00:13:40,400 Defender and Azure Sentinel. 202 00:13:40,400 --> 00:13:47,400 I haven't had time to play with this yet, but I hear from coworkers that is awesome 203 00:13:47,400 --> 00:13:50,280 resource. 204 00:13:50,280 --> 00:13:58,280 In addition, I wanted to talk about renewing release of Azure Sentinel sub process framework. 205 00:13:58,280 --> 00:14:05,200 As part of this podcast, we have talked about many of the threat protections and security 206 00:14:05,200 --> 00:14:13,000 operation cloud services that Microsoft has released and how they help streamline the 207 00:14:13,000 --> 00:14:16,280 protect the text and respond process. 208 00:14:16,280 --> 00:14:22,680 While these services provide many capabilities, I have seen instances where organizations 209 00:14:22,680 --> 00:14:30,120 do not take full advantage of all the capabilities provided by these services because their organizational 210 00:14:30,120 --> 00:14:37,760 process or procedures that they use in different in share with different team have not been 211 00:14:37,760 --> 00:14:39,260 updated. 212 00:14:39,260 --> 00:14:47,400 For example, I was hoping an organization that had Microsoft E5 threat protection suite, 213 00:14:47,400 --> 00:14:55,320 but then the tier one analysts were using some manual processes and other tools to construct 214 00:14:55,320 --> 00:15:00,920 reports that they were going to provide to their leadership. 215 00:15:00,920 --> 00:15:06,880 When I asked them why they did that, they mentioned that their processes dictated the 216 00:15:06,880 --> 00:15:11,400 tools and report format that they needed to use. 217 00:15:11,400 --> 00:15:19,280 Although our services provided more detail and more information, the only way for them 218 00:15:19,280 --> 00:15:30,760 to reproduce the reports they wanted in a fast manner was to use those other processes. 219 00:15:30,760 --> 00:15:38,360 In essence, this process framework that we released provides organization with an initial 220 00:15:38,360 --> 00:15:46,080 set of tasks that should be performed to ensure that issues like these are avoided. 221 00:15:46,080 --> 00:15:52,560 Also it helps the organization to know what are the processes that they need to be looking 222 00:15:52,560 --> 00:15:58,840 at, maybe readiness, hopefully take full advantage of the investment that they have made. 223 00:15:58,840 --> 00:16:04,840 There was a live presentation of the capabilities that RIN provided, but if you missed it, you 224 00:16:04,840 --> 00:16:12,760 could either search it in YouTube or I think the first week of July, RIN will be with us. 225 00:16:12,760 --> 00:16:17,200 So be in the lookout for our podcast during that week. 226 00:16:17,200 --> 00:16:24,680 Last, I wanted to mention about the IoT Hub Service APS support for Azure Active Directory 227 00:16:24,680 --> 00:16:26,520 based access control. 228 00:16:26,520 --> 00:16:35,480 Basically, you now can grant specific service API access permission to user service principles 229 00:16:35,480 --> 00:16:44,280 and manage identity from your Azure AD tenant using Azure RBAC or role-based access control 230 00:16:44,280 --> 00:16:49,280 to get started just grant roles with the new permissions. 231 00:16:49,280 --> 00:16:55,160 In the links that we are providing in our site, you will find some information about 232 00:16:55,160 --> 00:17:00,440 these building roles, permission and samples that you could use to implement this. 233 00:17:00,440 --> 00:17:03,680 Let's now turn our attention to our guest. 234 00:17:03,680 --> 00:17:05,040 We have Mini Walia. 235 00:17:05,040 --> 00:17:09,040 She's a senior program manager in the Azure Data Explorer product group. 236 00:17:09,040 --> 00:17:10,040 She's here to talk to us. 237 00:17:10,040 --> 00:17:13,880 Funnily enough, about Azure Data Explorer, many welcome to the podcast. 238 00:17:13,880 --> 00:17:16,200 Would you like to spend a moment explaining what you do at Microsoft? 239 00:17:16,200 --> 00:17:18,120 How long have you been with the company? 240 00:17:18,120 --> 00:17:19,240 Thanks, Michael. 241 00:17:19,240 --> 00:17:22,520 Thank you for having me on this podcast today. 242 00:17:22,520 --> 00:17:25,560 I'm really excited to be part of this series. 243 00:17:25,560 --> 00:17:30,320 My name is Mini Walia, and I'm working as a program manager with Azure Data Explorer 244 00:17:30,320 --> 00:17:33,280 product group in Microsoft. 245 00:17:33,280 --> 00:17:36,600 It's been almost four years with Microsoft. 246 00:17:36,600 --> 00:17:43,320 In this role, I work very closely with large enterprise customers to build secure and scalable 247 00:17:43,320 --> 00:17:48,520 big data analytics solutions that are mainly focused on telemetry data. 248 00:17:48,520 --> 00:17:55,000 I'm fortunate enough to work with some of the best Microsoft engineers to continuously 249 00:17:55,000 --> 00:18:01,040 evolve, improve and introduce new capabilities in our product. 250 00:18:01,040 --> 00:18:03,040 My first question is probably the most obvious one. 251 00:18:03,040 --> 00:18:07,040 I'll be honest, I had the same question when I first started learning about this. 252 00:18:07,040 --> 00:18:10,840 What is Azure Data Explorer? 253 00:18:10,840 --> 00:18:16,520 Azure Data Explorer, the short form we call it as ADX. 254 00:18:16,520 --> 00:18:21,760 It's a fully managed big data analytics platform. 255 00:18:21,760 --> 00:18:28,080 It's a distributed columnar store that is mainly purpose-built for real-time analytics 256 00:18:28,080 --> 00:18:30,360 over telemetry data. 257 00:18:30,360 --> 00:18:31,360 Let me clarify. 258 00:18:31,360 --> 00:18:37,400 When I say telemetry, it's a broader term that covers any type of logs, data coming from 259 00:18:37,400 --> 00:18:43,880 IoT devices, sensors, connected vehicles, or clickstream data, or it could be any type 260 00:18:43,880 --> 00:18:47,440 of events and user activities. 261 00:18:47,440 --> 00:18:53,440 Obviously, this is a product that is designed to ingest significant amounts of data. 262 00:18:53,440 --> 00:18:55,960 What runs underneath this? 263 00:18:55,960 --> 00:18:58,880 It's a Microsoft's proprietary database. 264 00:18:58,880 --> 00:19:06,240 Think of it like an append-only analytical database, which is a bit different from the 265 00:19:06,240 --> 00:19:13,000 typical general-purpose databases or transactional databases we have, for example, SQL DB or 266 00:19:13,000 --> 00:19:15,440 Cosmos DB. 267 00:19:15,440 --> 00:19:21,600 In a sense, I would say it's not really meant for transactional scenarios or frequent update 268 00:19:21,600 --> 00:19:23,200 to lead scenarios. 269 00:19:23,200 --> 00:19:30,560 It's built for analytical append-only workloads to build low latency, high throughput, near 270 00:19:30,560 --> 00:19:32,800 real-time analytics dashboards. 271 00:19:32,800 --> 00:19:38,880 Mini, when we were talking about this before we started recording the podcast, we were 272 00:19:38,880 --> 00:19:43,440 talking about the scenarios for using ADX. 273 00:19:43,440 --> 00:19:48,400 There's roughly maybe about three main scenarios you see at a high level. 274 00:19:48,400 --> 00:19:51,400 Could you talk some more about those? 275 00:19:51,400 --> 00:19:52,400 Sure. 276 00:19:52,400 --> 00:19:59,800 There are three broader scenarios which are meant for using Azure Data Explorer or, as 277 00:19:59,800 --> 00:20:02,400 I said, ADX short form. 278 00:20:02,400 --> 00:20:05,480 The first one is around telemetry analytics. 279 00:20:05,480 --> 00:20:08,840 That is what it is purposely built for. 280 00:20:08,840 --> 00:20:14,960 It's mainly focused on log analytics and time series analytics scenarios wherein you 281 00:20:14,960 --> 00:20:23,000 can do interactive analytics and Hock explorations of data and build near real-time dashboards. 282 00:20:23,000 --> 00:20:28,560 The second broader scenario would be around advanced analytics. 283 00:20:28,560 --> 00:20:37,120 ADX powers data science and machine learning workloads with a lot of native capabilities 284 00:20:37,120 --> 00:20:46,040 for pattern recognitions, forecasting, anomaly detections, those advanced scenarios. 285 00:20:46,040 --> 00:20:54,840 The third scenario is wherein a lot of customers and ISVs, they build single or multi-tenant 286 00:20:54,840 --> 00:20:59,720 SaaS solutions using ADX similar to what Microsoft has done. 287 00:20:59,720 --> 00:21:07,040 For example, Microsoft has built various SaaS solutions on top of ADX in different domains, 288 00:21:07,040 --> 00:21:11,840 like we have got in the monitoring domain, we have Azure Monitor. 289 00:21:11,840 --> 00:21:17,160 In the security domain, we have got Azure Sentinel, Security Center, Advanced Thread 290 00:21:17,160 --> 00:21:18,720 Protection. 291 00:21:18,720 --> 00:21:25,920 Same way, we have got different products in the IoT domain, time series insights, Play 292 00:21:25,920 --> 00:21:28,520 Fab in the gaming domain, so on and so forth. 293 00:21:28,520 --> 00:21:31,680 There are heaps of products which are built on top of it. 294 00:21:31,680 --> 00:21:40,000 In nutshell, three key broader scenarios, telemetry analytics, advanced analytics, and 295 00:21:40,000 --> 00:21:42,680 SaaS solutions which you can build on top of it. 296 00:21:42,680 --> 00:21:46,480 Well, I'm going to dig into one of the things you mentioned there. 297 00:21:46,480 --> 00:21:53,160 Obviously, everybody who listens to this podcast knows I have to bring it in somehow, but you 298 00:21:53,160 --> 00:21:55,440 mentioned log analytics. 299 00:21:55,440 --> 00:21:56,440 Could you explain that? 300 00:21:56,440 --> 00:22:01,200 Of course, log analytics sits underneath Azure Sentinel, but what's the difference 301 00:22:01,200 --> 00:22:05,520 between ADX and log analytics? 302 00:22:05,520 --> 00:22:12,440 Log analytics is basically a SaaS solution that's built on top of ADX. 303 00:22:12,440 --> 00:22:20,440 ADX is a PaaS solution, the underlying platform on which log analytics is built. 304 00:22:20,440 --> 00:22:27,440 Log analytics has got out of the box capabilities or I would say domain knowledge in the infrastructure 305 00:22:27,440 --> 00:22:30,600 monitoring space. 306 00:22:30,600 --> 00:22:39,040 As ADX is an underlying platform, so it brings in value in terms of providing full flexibility 307 00:22:39,040 --> 00:22:41,240 and full control on data. 308 00:22:41,240 --> 00:22:45,960 When I say data, it could be the management of data, data schema, so the customers get 309 00:22:45,960 --> 00:22:50,360 full access to this underlying platform. 310 00:22:50,360 --> 00:22:59,000 You can do powerful native analytics on top of this telemetry and time series data. 311 00:22:59,000 --> 00:23:04,360 It's a very cost effective platform. 312 00:23:04,360 --> 00:23:11,640 To continue Sarah's thought, these are obviously two different products as you data explorer 313 00:23:11,640 --> 00:23:13,560 and log analytics. 314 00:23:13,560 --> 00:23:16,560 Do they bring different value propositions to customers? 315 00:23:16,560 --> 00:23:17,560 Absolutely. 316 00:23:17,560 --> 00:23:26,800 As I said, the key value proposition ADX brings in is in form of bringing in full flexibility, 317 00:23:26,800 --> 00:23:33,320 full control and customers can go to the granular level of managing their data. 318 00:23:33,320 --> 00:23:40,680 For example, they can even manage till the role level security of their data. 319 00:23:40,680 --> 00:23:44,920 The best part of ADX is its price performance ratio. 320 00:23:44,920 --> 00:23:48,920 It can be used as a hybrid solution. 321 00:23:48,920 --> 00:23:50,160 Customers mix and match. 322 00:23:50,160 --> 00:23:57,080 They can use log analytics as well as ADX, a hybrid solution to get the best of both 323 00:23:57,080 --> 00:23:58,080 the worlds. 324 00:23:58,080 --> 00:24:07,160 Think of it like Microsoft has built SaaS solutions on top of ADX and we are providing 325 00:24:07,160 --> 00:24:12,400 full transparency and full control even on the underlying platform. 326 00:24:12,400 --> 00:24:15,800 You can customize the way you need. 327 00:24:15,800 --> 00:24:18,000 I think that's a really important point actually. 328 00:24:18,000 --> 00:24:22,160 I really want to make sure I listen to understand this. 329 00:24:22,160 --> 00:24:23,720 We have this tool, log analytics. 330 00:24:23,720 --> 00:24:29,880 We have tools like Sentinel and so on that feed into log analytics. 331 00:24:29,880 --> 00:24:35,440 But ultimately, if you want absolute access to the low level data for your own analysis 332 00:24:35,440 --> 00:24:38,920 work, then that's provided through Azure Data Explorer. 333 00:24:38,920 --> 00:24:41,320 Did I get that right? 334 00:24:41,320 --> 00:24:42,320 Absolutely. 335 00:24:42,320 --> 00:24:43,320 Spot on, Michael. 336 00:24:43,320 --> 00:24:45,320 You got it absolutely right. 337 00:24:45,320 --> 00:24:47,920 I have another question, Mini. 338 00:24:47,920 --> 00:24:51,640 You mentioned that ADX is something that we use internally. 339 00:24:51,640 --> 00:24:55,640 Can you talk more about what Microsoft does with ADX? 340 00:24:55,640 --> 00:24:57,920 Sure. 341 00:24:57,920 --> 00:25:04,480 As I said, ADX is used internally, heavily, and that's how we started. 342 00:25:04,480 --> 00:25:08,880 Let me give you a bit of a background around the journey, how we started. 343 00:25:08,880 --> 00:25:16,480 In 2015, Microsoft started using ADX internally, heavily for collecting its telemetry data 344 00:25:16,480 --> 00:25:26,520 from a lot of services, including Power BI ecosystem, SQL servers, Windows, and there 345 00:25:26,520 --> 00:25:29,160 are many other systems that are sending its telemetry. 346 00:25:29,160 --> 00:25:34,600 Just to give you an idea, for example, think of it like every VM in every Azure Data Center 347 00:25:34,600 --> 00:25:38,320 globally is sending its telemetry to this platform. 348 00:25:38,320 --> 00:25:45,120 We are collecting more than 40 petabytes of data every day onto this platform. 349 00:25:45,120 --> 00:25:52,120 In total, it's dealing with 2.5 plus exabytes of data. 350 00:25:52,120 --> 00:25:54,600 That's the scale it works at. 351 00:25:54,600 --> 00:26:01,920 You can imagine how Microsoft is using it internally for handling its massive petabyte 352 00:26:01,920 --> 00:26:04,240 scale of telemetry. 353 00:26:04,240 --> 00:26:09,960 After seeing significant success internally, we made it available to our external customers 354 00:26:09,960 --> 00:26:15,240 and the service went GA for the external customers in 2019. 355 00:26:15,240 --> 00:26:16,480 Cool. 356 00:26:16,480 --> 00:26:18,280 It's very impressive stuff. 357 00:26:18,280 --> 00:26:24,680 If any of our listeners are thinking, oh my God, I want to use ADX, how would a customer 358 00:26:24,680 --> 00:26:28,760 go about starting to use ADX? 359 00:26:28,760 --> 00:26:30,440 Sure. 360 00:26:30,440 --> 00:26:31,840 It's super easy. 361 00:26:31,840 --> 00:26:35,640 It's simply just like any other service on Azure. 362 00:26:35,640 --> 00:26:41,280 You go to the portal, create your Azure Data Explorer cluster, and you can provision cluster 363 00:26:41,280 --> 00:26:45,400 either via portal or through the ARM templates. 364 00:26:45,400 --> 00:26:46,400 That's it. 365 00:26:46,400 --> 00:26:53,440 Once the cluster is created, you create your databases, tables underneath, and you can access 366 00:26:53,440 --> 00:26:56,880 the cluster via the tools we provide. 367 00:26:56,880 --> 00:27:02,640 There is an ADX web UI, which is very easy and user-friendly tool. 368 00:27:02,640 --> 00:27:08,880 You can use to ingest the data, for example, with a single click, run your interactive 369 00:27:08,880 --> 00:27:11,680 analytics queries, build your dashboards. 370 00:27:11,680 --> 00:27:15,840 All those capabilities are baked into that ADX web UI. 371 00:27:15,840 --> 00:27:21,000 We do have Thick Client as well, but most of the customers, they prefer to start with 372 00:27:21,000 --> 00:27:23,680 the ADX web UI. 373 00:27:23,680 --> 00:27:27,280 When would customers use Log Analytics and ADX together? 374 00:27:27,280 --> 00:27:31,520 Are there any scenarios that come to mind? 375 00:27:31,520 --> 00:27:37,160 A lot of customers who are building hybrid solutions using both these platforms, the 376 00:27:37,160 --> 00:27:43,240 key scenario is when customers are dealing with massive amounts of data. 377 00:27:43,240 --> 00:27:50,480 For example, Microsoft is using ADX for handling petabyte scale of telemetry data. 378 00:27:50,480 --> 00:27:55,760 Same way when customers want a viable, scalable, and cost-effective solution. 379 00:27:55,760 --> 00:28:04,640 Plus, when some of the requirements need customizations or full control on data, that's when they 380 00:28:04,640 --> 00:28:05,640 use ADX. 381 00:28:05,640 --> 00:28:13,600 ADX becomes a centralized repository for all of their data. 382 00:28:13,600 --> 00:28:22,280 They do take value which Log Analytics brings in in the monitoring space, or Sentinel brings 383 00:28:22,280 --> 00:28:25,040 in in the security domain. 384 00:28:25,040 --> 00:28:30,400 They use ADX even for longer retention of data. 385 00:28:30,400 --> 00:28:39,120 For example, they use Sentinel for 90 days of immediate analysis and detection of their 386 00:28:39,120 --> 00:28:40,120 data. 387 00:28:40,120 --> 00:28:48,400 They route data past 90 days to ADX for longer-term retention for audit or compliance or other 388 00:28:48,400 --> 00:28:50,760 reasons. 389 00:28:50,760 --> 00:28:53,200 That's another scenario which they use it for. 390 00:28:53,200 --> 00:28:56,960 It's silly me asking the question because I've been working with a customer in this 391 00:28:56,960 --> 00:28:59,720 exact space just recently. 392 00:28:59,720 --> 00:29:03,120 The conversation went like this, hey, it's going to cost us a lot of money to store all 393 00:29:03,120 --> 00:29:05,240 this log data. 394 00:29:05,240 --> 00:29:06,240 What log should we store? 395 00:29:06,240 --> 00:29:10,560 Unfortunately, when it comes to security information, you want to store everything because you never 396 00:29:10,560 --> 00:29:14,120 know a few months down the track an incident may happen. 397 00:29:14,120 --> 00:29:18,360 If you didn't have the log, then you can't do the forensics. 398 00:29:18,360 --> 00:29:25,200 They're basically ingesting absolutely everything, not just a cost perspective, but also from 399 00:29:25,200 --> 00:29:31,320 a minimal technical depth perspective, they're looking at using Azure Data Explorer, exactly 400 00:29:31,320 --> 00:29:32,320 for this. 401 00:29:32,320 --> 00:29:36,200 They're just ingesting absolutely everything, putting it into Azure Data Explorer, and by 402 00:29:36,200 --> 00:29:42,400 far the cheapest way they found for storing just these incredible amounts of data. 403 00:29:42,400 --> 00:29:45,800 Absolutely, yes. 404 00:29:45,800 --> 00:29:48,560 There are a lot of features which are available. 405 00:29:48,560 --> 00:29:53,800 For example, we do support cross cluster querying. 406 00:29:53,800 --> 00:30:00,560 Customers can ingest data into Sentinel, which uses log analytics workspace under the hood. 407 00:30:00,560 --> 00:30:05,600 ADX is the underlying platform for both of these products. 408 00:30:05,600 --> 00:30:09,320 They can build their federated queries. 409 00:30:09,320 --> 00:30:15,480 They can enrich the data by building the federated queries via these cross cluster querying capabilities. 410 00:30:15,480 --> 00:30:22,040 They can, for example, create dashboards where they can bring in the data from both these 411 00:30:22,040 --> 00:30:25,720 platforms without moving data here and there. 412 00:30:25,720 --> 00:30:30,200 There is a lot of flexibility, as I said. 413 00:30:30,200 --> 00:30:34,680 Is there anything you've talked about a lot of cool tools and features, but is there anything 414 00:30:34,680 --> 00:30:42,160 that's coming up that's exciting in ADX LAN that you're able to tell us about? 415 00:30:42,160 --> 00:30:48,600 We keep evolving, improving, and keep adding a lot of features in our product. 416 00:30:48,600 --> 00:30:57,120 The hottest thing which is coming up is the integration of ADX within Azure Synapse Analytics. 417 00:30:57,120 --> 00:31:03,080 Azure Synapse Analytics is a single stop shop solution for the customers for their big data 418 00:31:03,080 --> 00:31:04,480 analytics solutions. 419 00:31:04,480 --> 00:31:11,720 ADX integration within Synapse fills in the gaps for managing and for dealing with the 420 00:31:11,720 --> 00:31:19,440 log analytics and time series analytics scenarios within a single umbrella. 421 00:31:19,440 --> 00:31:21,160 That's what we are working on. 422 00:31:21,160 --> 00:31:23,560 It's in private preview at the moment. 423 00:31:23,560 --> 00:31:28,080 If anyone is interested, they can reach out to us. 424 00:31:28,080 --> 00:31:36,200 Another cool thing we are working on is contributing onto the open source solution, Telegraph. 425 00:31:36,200 --> 00:31:42,640 Telegraph is an open source solution for collecting metrics, and we are building ADX output plug-in 426 00:31:42,640 --> 00:31:50,280 for Telegraph so customers can customize and ingest data using n number of input plugins 427 00:31:50,280 --> 00:31:53,960 that are available within Telegraph space. 428 00:31:53,960 --> 00:31:57,080 These are two cool things relevant to the monitoring space. 429 00:31:57,080 --> 00:32:02,400 Minnie, if you've got one thought to leave our listeners with, we ask this to every single 430 00:32:02,400 --> 00:32:06,600 guest we have on the podcast, what would it be? 431 00:32:06,600 --> 00:32:13,080 So, Sarah, I would sum it up by sharing a quote from one of our customers. 432 00:32:13,080 --> 00:32:18,120 So when we made ADX available for our external customers after seeing significant success 433 00:32:18,120 --> 00:32:25,960 internally, the customer after using ADX, they shared their experience by saying ADX 434 00:32:25,960 --> 00:32:29,360 is Microsoft's best kept secret. 435 00:32:29,360 --> 00:32:35,680 It's very mature and I would say it's a battle-tested platform over petabyte scale. 436 00:32:35,680 --> 00:32:40,760 As I mentioned, how Microsoft uses it for ingesting 40 plus petabytes of data every 437 00:32:40,760 --> 00:32:42,280 day. 438 00:32:42,280 --> 00:32:50,000 And the last thing I would say is it's super easy, very user-friendly database, and it 439 00:32:50,000 --> 00:32:57,840 comes up with tools and, for example, ADX Web UI I talked about and we support Kusto 440 00:32:57,840 --> 00:33:04,280 Query language, which is, again, very, very simple and user-friendly and SQL is also supported. 441 00:33:04,280 --> 00:33:10,880 I would encourage all the users to try it out and we are always keen to hear your feedback 442 00:33:10,880 --> 00:33:13,360 to improve our products. 443 00:33:13,360 --> 00:33:15,360 And with that, let's bring this episode to an end. 444 00:33:15,360 --> 00:33:17,400 Minnie, thank you so much for joining us this week. 445 00:33:17,400 --> 00:33:20,980 Just for our listeners out there, Minnie isn't feeling very well, so thank you so much, 446 00:33:20,980 --> 00:33:24,320 Minnie, for coming along and joining us. 447 00:33:24,320 --> 00:33:27,680 To our listeners out there, I hope you found this podcast useful. 448 00:33:27,680 --> 00:33:28,840 I know I definitely did. 449 00:33:28,840 --> 00:33:31,880 I always learn something and this is certainly no exception. 450 00:33:31,880 --> 00:33:34,240 So to everyone out there, stay safe and we'll see you next time. 451 00:33:34,240 --> 00:33:37,400 Thanks for listening to the Azure Security Podcast. 452 00:33:37,400 --> 00:33:44,200 You can find show notes and other resources at our website azsecuritypodcast.net. 453 00:33:44,200 --> 00:33:49,360 If you have any questions, please find us on Twitter at AzureSecPod. 454 00:33:49,360 --> 00:33:59,520 Music is from ccmixter.com and licensed under the Creative Commons license.