WEBVTT 00:00:00.000 --> 00:00:05.140 You are listening to Support Engineering Weekly. 00:00:05.140 --> 00:00:10.420 You know, it is incredibly common for engineers, architects, I mean, even really seasoned 00:00:10.420 --> 00:00:15.420 developers to treat Microsoft data versus just like a simple backend database. 00:00:15.420 --> 00:00:16.940 Oh, absolutely. It happens all the time. 00:00:16.940 --> 00:00:20.740 Right. You spin up a power platform application. You realize you need a place to just kind of 00:00:20.740 --> 00:00:23.660 dump the user data and boom, you just create some table. 00:00:23.660 --> 00:00:25.860 Yeah, you just treat it like any other SQL bucket. 00:00:25.860 --> 00:00:28.940 Exactly. It feels very familiar, very standard. 00:00:28.940 --> 00:00:34.360 But assuming it is just a relational SQL bucket where rows and columns just, you know, 00:00:34.360 --> 00:00:38.360 live in isolation, yeah, that assumption is an absolute enterprise trap. 00:00:38.360 --> 00:00:44.000 It really is. I mean, treating data versus like a basic database completely ignores the 00:00:44.000 --> 00:00:46.440 reality of what the system actually is mechanically. 00:00:46.440 --> 00:00:47.940 Yeah, it's so much more than that. 00:00:47.940 --> 00:00:54.300 It is a massive, multi-layered, managed application data platform. Like the moment you try to 00:00:54.300 --> 00:00:57.820 force it to behave like a standard isolated database. 00:00:57.820 --> 00:01:03.060 Using raw queries and stuff. Yeah, writing raw queries bypassing the API or just ignoring 00:01:03.060 --> 00:01:07.300 its life cycle rules entirely, things start breaking down catastrophically at scale. 00:01:07.300 --> 00:01:09.500 I'm Alex, by the way. And I'm Maya. 00:01:09.500 --> 00:01:14.539 And we want to sincerely thank you for joining today's deep dives into our source material, 00:01:14.539 --> 00:01:18.980 because our mission today is to completely dissect the dataverse architecture. 00:01:18.980 --> 00:01:20.740 We really want to get into the weeds here. 00:01:20.740 --> 00:01:27.820 We do. We are going to look at how metadata, security, API's capacity, and application 00:01:27.820 --> 00:01:31.179 life cycle management all interact under the hood. 00:01:31.179 --> 00:01:36.220 Because once you understand how these specific gears turned together, you can actually engineer 00:01:36.220 --> 00:01:38.460 a predictable governed platform. 00:01:38.460 --> 00:01:41.940 Right. Rather than just kind of crossing your fingers and hoping your apps don't crash 00:01:41.940 --> 00:01:44.300 when, you know, user traffic spikes. 00:01:44.300 --> 00:01:48.699 Yeah, hope is not a great engineering strategy to really understand why this platform is 00:01:48.740 --> 00:01:53.700 fundamentally different from a standalone SQL server. We have to trace what actually 00:01:53.700 --> 00:01:55.860 happens when a request hits the system. 00:01:55.860 --> 00:02:00.620 Okay, let's unpack this. Because the architecture here is often described in our sources as 00:02:00.620 --> 00:02:05.260 a six layer kick. Right. That's right. When an application saves a record, it is not 00:02:05.260 --> 00:02:09.220 a straight line to a hard drive. It doesn't just, you know, write to disk. 00:02:09.220 --> 00:02:15.820 So think of dataverse, not as a simple storage unit, but as a, like a high security corporate 00:02:15.820 --> 00:02:19.820 headquarters, you don't just walk in off the street and drop a file on a desk in the 00:02:19.820 --> 00:02:21.859 basement vault. No, they would tackle you. 00:02:21.859 --> 00:02:25.739 Exactly. First you have to go through the front desk. Those are your API's. Then you 00:02:25.739 --> 00:02:29.579 have to get badge clearance, which is the security layer. Right. Checking if you're even allowed 00:02:29.579 --> 00:02:35.299 in the building. Yeah. After that, you have to follow company policy, meaning the platform 00:02:35.299 --> 00:02:40.180 services and business logic layers have to approve your action, which is a whole other 00:02:40.180 --> 00:02:45.220 set of checks. Right. And only after you clear all those hurdles, does your file actually 00:02:45.220 --> 00:02:48.460 take the elevator down to the vault, which is the storage layer. 00:02:48.460 --> 00:02:52.220 That is a highly accurate way to visualize it. And covering all of that, you have the 00:02:52.220 --> 00:02:56.460 experience layer at the very top. Oh, like the user interfaces. Yeah. This includes your 00:02:56.460 --> 00:03:02.260 canvas apps, your model driven apps, and your co-pilot agents. You also have the integration 00:03:02.260 --> 00:03:08.460 and analytics layer, which maps the data outward to other enterprise systems. Plus, there is 00:03:08.460 --> 00:03:14.020 a governance and operations layer overlaying the entire stack. So when a JSON payload comes 00:03:14.020 --> 00:03:20.020 in from say, a custom app or power automate flow, it has to physically pass through that 00:03:20.020 --> 00:03:24.940 API access layer. It has to run the gauntlet. Exactly. It navigates the platform services, 00:03:24.940 --> 00:03:29.700 the metadata checks, the security engine, all of that before a single byte is ever committed 00:03:29.700 --> 00:03:34.660 to storage, which means we completely have to change how we think about making system 00:03:34.660 --> 00:03:39.620 updates. Because if you tweak something in dataverse, you aren't just like altering 00:03:39.620 --> 00:03:43.420 a table definition in a vacuum. What's fascinating here is why this matters 00:03:43.420 --> 00:03:48.220 operationally. A schema change in this environment is not just a database tweet because of all 00:03:48.220 --> 00:03:52.820 those layers. Yes, because of those interconnected layers, adding a column or changing a relationship 00:03:52.820 --> 00:03:57.980 alters the behavior of applications, API's and security almost instantly instantly. Yeah, 00:03:57.980 --> 00:04:02.299 you modify a table and suddenly the API latency might shift because of how the data is being 00:04:02.299 --> 00:04:07.779 routed. Or a synchronous plugin firing at the platform services layer might behave completely 00:04:07.779 --> 00:04:12.700 differently because the data structure it expects just changed. Wow. So if a single request 00:04:12.700 --> 00:04:17.539 has to pass through the front desk, the security gate, and the policy checks, how does the 00:04:17.539 --> 00:04:22.620 system actually know what rules to apply to that specific request in real time? 00:04:22.620 --> 00:04:28.180 Well, the architecture uses metadata as the glue holding all these layers together. 00:04:28.180 --> 00:04:33.900 But wait, isn't metadata just data about data? Like if I right click a photo on my laptop, 00:04:33.900 --> 00:04:38.980 the metadata just the data file was created or the camera model used. Yeah. Why is it 00:04:38.980 --> 00:04:40.940 treated as such a big deal in this architecture? 00:04:41.260 --> 00:04:45.540 I mean, in a traditional file system, you absolutely write metadata is mostly just descriptive 00:04:45.540 --> 00:04:50.980 tags sitting passively on a file. But data versus fundamentally a metadata driven architecture. 00:04:50.980 --> 00:04:56.580 It's not passive. Not at all. Here, metadata is active and structural. It dictates engine 00:04:56.580 --> 00:05:00.700 behavior. What does that mean? Practically, it means the table metadata defines not just 00:05:00.700 --> 00:05:05.700 what data type is stored in a column, but how records are created, what specific actions 00:05:05.700 --> 00:05:10.900 can be performed on them by the API. Oh, wow. Yeah. And how forms and views are 00:05:10.900 --> 00:05:15.260 dynamically generated for the user interface and even how the cascading security rules 00:05:15.260 --> 00:05:21.700 behave. It is the active rule book that the platform services layer reads continuously. 00:05:21.700 --> 00:05:28.099 So it's not just describing the table. The metadata is actively generating the API endpoints 00:05:28.099 --> 00:05:33.380 and the application behavior for that table on the fly. Exactly that. The source material 00:05:33.380 --> 00:05:38.739 explicitly refers to the schema as a platform contract, not just a database structure. 00:05:38.740 --> 00:05:43.540 A contract. I like that. When you define a table in Dataverse, you are signing a strict 00:05:43.540 --> 00:05:48.340 contract with the rest of the stack about how that entity will behave across every single layer 00:05:48.340 --> 00:05:52.580 we just talked about, which totally explains why the architectural guidance is so strict about 00:05:52.580 --> 00:05:57.380 using standard out of the box tables whenever they fit the business concept. Oh, absolutely. 00:05:57.380 --> 00:06:01.860 Like if you need a table to track accounts or contacts, the recommendation is to use the built 00:06:01.860 --> 00:06:06.260 in ones rather than building your own custom, I don't know, my company accounts table from scratch. 00:06:06.260 --> 00:06:11.380 But why? If I build it myself, I control all the columns. Well, you do control the column, 00:06:11.380 --> 00:06:16.340 sure, but you lose the platform contract. You really have to question your assumptions here 00:06:16.340 --> 00:06:22.820 about control versus technical debt. Okay, so if you build unnecessary custom tables, 00:06:22.820 --> 00:06:28.180 instead of using the standard ones, you are throwing away thousands of hours of Microsoft's 00:06:28.180 --> 00:06:32.980 engineering. Right, because they already did the work. Exactly. The standard tables already have 00:06:32.980 --> 00:06:39.620 highly optimized relationships, complex security models, and specific API behaviors hardwired 00:06:39.620 --> 00:06:45.780 into the platform's engine. So when you reinvent the wheel, you lose all that out of the box optimization. 00:06:45.780 --> 00:06:51.380 And you take on the burden of maintaining a much more fragile custom contract that frankly might 00:06:51.380 --> 00:06:56.180 not perform as well under heavy load. Right. So assuming we've honored the metadata contract and 00:06:56.180 --> 00:07:00.660 we aren't reinventing the wheel, the data finally makes it down to the vault, the storage layer, 00:07:00.660 --> 00:07:04.340 the bottom of the cake. Right. But again, the trap we mentioned earlier is thinking 00:07:04.340 --> 00:07:09.140 data versus just tosses everything into one massive relational SQL database. It absolutely 00:07:09.140 --> 00:07:13.540 doesn't. And assuming it does, we'll ruin your capacity planning. So what is it actually doing 00:07:13.540 --> 00:07:19.300 down there? Dataverse acts as a router. It sends data to highly specialized storage engines under 00:07:19.300 --> 00:07:24.740 the hood based on the actual workload. Like different types of vaults. Yeah. For your transactional, 00:07:24.740 --> 00:07:31.700 relational business, beta like accounts, invoices, complex relationships, it uses standard or custom 00:07:31.700 --> 00:07:36.740 tables, which are backed by Azure SQL. Okay, that makes sense for standard business logic. 00:07:36.740 --> 00:07:44.020 But for high volume, bursty semi structured workloads like IoT sensor data or interaction logs, 00:07:44.020 --> 00:07:49.540 Dataverse offers elastic tables. And those are actually backed by Azure Cosmos DB. 00:07:49.540 --> 00:07:52.820 And this is a huge trap I see in enterprise deployments all the time. 00:07:53.380 --> 00:07:58.180 People treat elastic tables like they're just a toggle switch for faster standard tables. 00:07:58.180 --> 00:08:01.940 Oh, I've seen that too. They look at their data. Think, Oh, I have a lot of rows coming in quickly. 00:08:01.940 --> 00:08:07.700 I'll just use the elastic table for better performance. Why is applying an elastic table to a standard 00:08:07.700 --> 00:08:13.620 business problem? Such a dangerous architectural move. Because Cosmos DB, which powers those 00:08:13.620 --> 00:08:19.140 elastic tables operates on completely different database principles than a traditional relational 00:08:19.219 --> 00:08:24.979 SQL database. Right. It's not SQL. No, when we talk about standard SQL tables, we rely on 00:08:24.979 --> 00:08:31.539 AC transactions, atomicity, consistency, isolation, durability. Right. The golden rules of SQL. 00:08:31.539 --> 00:08:38.020 Exactly. If you save an account in a contact at the exact same time as part of a single transaction, 00:08:38.020 --> 00:08:43.380 ACD guarantees that either both save successfully or neither saves. It's perfectly consistent. 00:08:43.380 --> 00:08:48.580 But Cosmos DB doesn't play by those exact same rules. Right. Cosmos DB is designed for a 00:08:48.580 --> 00:08:54.420 horizontal scale and speed, often relying on eventual consistency. The eventual consistency. 00:08:54.420 --> 00:08:59.380 So not instant. Right. The data is partitioned differently. If you try to build a highly 00:08:59.380 --> 00:09:04.500 relational ACD dependent financial business process on top of an elastic table, just because you wanted 00:09:04.500 --> 00:09:10.740 the rights to be, you know, fast, you are going to encounter massive architectural friction. 00:09:10.740 --> 00:09:15.540 It sounds messy. It is. The data might not be instantly available across all nodes. The millisecond 00:09:15.620 --> 00:09:19.620 is written. You have to match the data model mechanically to the specific workload. 00:09:19.620 --> 00:09:23.140 And the mechanical complexity that storage directly impacts capacity planning to 00:09:23.140 --> 00:09:29.219 is significantly because capacity and dataverse isn't just a single flat metric of how many gigabytes 00:09:29.219 --> 00:09:35.140 do I have left? The source material breaks it into three distinct buckets. Yes. Database capacity 00:09:35.140 --> 00:09:40.900 for your structured data and active metadata file capacity for heavy things like PDFs and image 00:09:40.980 --> 00:09:47.380 attachments and log capacity for audit and trace data. If we connect this to the bigger picture, 00:09:47.380 --> 00:09:53.140 treating those three distinct buckets as one single storage problem leads to massive cost 00:09:53.140 --> 00:09:58.260 overruns. I can imagine. Let's look closely at auditing actually because that is where organizations 00:09:58.260 --> 00:10:02.579 really bleed money. It's a huge issue. Say a company migrates their legacy system to 00:10:02.579 --> 00:10:08.420 dataverse and the internal compliance team issues a mandate. We need to audit everything. Oh no. 00:10:08.420 --> 00:10:13.459 Right. Turn on auditing for every single table, every column and track every single read and write. 00:10:13.459 --> 00:10:16.819 I can hear the billing alarms going off in my head already. You should because all of that 00:10:16.819 --> 00:10:21.699 tracking data flows directly into that third bucket log capacity. Right. Mechanically, 00:10:21.699 --> 00:10:26.339 every time a user simply opens a record to look at it, the system has to generate a separate log 00:10:26.339 --> 00:10:32.900 entry and write it to storage just for opening it. Yes. If you blindly turn on comprehensive 00:10:32.980 --> 00:10:39.459 auditing for the entire system, your log capacity explodes exponentially. It will dwarf your actual 00:10:39.459 --> 00:10:44.819 business data in a matter of weeks. That's insane. And that directly impacts your compliance operations 00:10:44.819 --> 00:10:51.540 and your monthly storage costs. Exactly. Audit design is fundamentally a capacity and cost decision, 00:10:51.540 --> 00:10:56.579 not just a checkbox on a security document. You have to forecast the growth curve for database, 00:10:56.579 --> 00:11:02.100 file and log capacity completely separately. Okay. So we have data landing in these highly 00:11:02.100 --> 00:11:07.860 optimized storage buckets and we've got audit logs generating massive capacity to monitor activity. 00:11:07.860 --> 00:11:12.740 But monitoring after the fact isn't actual protection. No, it's just a record of what happened. 00:11:12.740 --> 00:11:17.780 Right. The real question is how the system guards the front door and the vault at the exact same time. 00:11:17.780 --> 00:11:21.940 Security in this architecture is cumulative. It's not a single toggle switch where you are either 00:11:21.940 --> 00:11:27.380 an admin or a guest. It's not binary. It is a gauntlet of seven distinct layers that every single 00:11:27.380 --> 00:11:31.939 request has to survive sequentially to gain effective access to a piece of data. 00:11:31.939 --> 00:11:37.700 Let's run through them because of the sheer depth of this is wild. First, the request hits 00:11:37.700 --> 00:11:44.100 Microsoft Antra ID for baseline authentication. Right. Just to prove you are who you say you are. 00:11:44.100 --> 00:11:50.100 Right. Assuming the system knows who you are, it checks licensing to see if you are legally 00:11:50.100 --> 00:11:54.740 allowed to use the platform. Then it checks the environment boundary. Are you allowed in this 00:11:54.740 --> 00:11:59.540 specific sandbox? And that's just to get into the outer perimeter. Exactly. Once you were in the 00:11:59.540 --> 00:12:05.380 environment, the system checks your specific security roles. But roles are enough on their own. 00:12:05.380 --> 00:12:11.060 Right. There's more. It then checks your business unit or team affiliation to see where you sit 00:12:11.060 --> 00:12:16.500 in the company hierarchy. Then it evaluates ownership and sharing. Do you actually own the 00:12:16.500 --> 00:12:20.420 record you were trying to read or has someone explicitly shared it with you? And finally, 00:12:20.420 --> 00:12:25.300 even if you pass all of that, it checks column security. Yes. Like you might be able to see the 00:12:25.300 --> 00:12:30.740 employee record, but the system strips out their social security number column before sending it 00:12:30.740 --> 00:12:34.740 to your screen. It's exactly like traveling internationally. That's an excellent way to 00:12:34.740 --> 00:12:39.380 visualize the mechanics of it. Think about it. You need your passport just to get through the 00:12:39.380 --> 00:12:45.300 airport doors. That's Antra ID. You need a valid plane ticket. That's your dataverse license. 00:12:45.300 --> 00:12:50.500 You need clearance to enter a specific terminal. That's your environment. Right. Right. And then 00:12:50.500 --> 00:12:56.020 you need your specific seat assignment on the plane. Those are your roles, business units, 00:12:56.020 --> 00:13:02.099 and ownership rules. If you are missing just one of those pieces of paper, you aren't flying. 00:13:02.099 --> 00:13:07.939 The access is entirely denied. And that cumulative nature is exactly why application access does 00:13:07.939 --> 00:13:12.420 not automatically equal data access. Well, that's a big one. This confuses developers new to the 00:13:12.420 --> 00:13:17.540 platform all the time. Just because a user has the link to open a power app and the app loads on 00:13:17.540 --> 00:13:21.860 their phone doesn't mean they can see the records inside it because they still have to pass the data 00:13:21.860 --> 00:13:27.219 security column. Precisely. If their business unit restricts them or they lack the specific 00:13:27.219 --> 00:13:32.420 record ownership, the system will render the app interface perfectly, but the galleries will just 00:13:32.420 --> 00:13:37.459 be blank. The data is blocked at the platform level. The architecture warning from our source 00:13:37.460 --> 00:13:42.740 material is incredibly clear on this point. Security must be designed before a large-scale 00:13:42.740 --> 00:13:49.860 data migration. I've seen the aftermath of ignoring that warning. You import like five million records 00:13:49.860 --> 00:13:54.180 and just assign them all to a generic system admin account to get it done quickly. Oh, the 00:13:54.180 --> 00:13:59.460 classic quick fix. But then you realize you need granular security. So you try to retrofit those 00:13:59.460 --> 00:14:04.420 seven layers of cumulative security afterward. It is an absolute nightmare to untangle because 00:14:04.420 --> 00:14:10.500 the system has to recalculate the cascading access rules for millions of rows is practically 00:14:10.500 --> 00:14:15.699 impossible to retrofit without causing severe operational downtime. The system just locks up. 00:14:15.699 --> 00:14:20.979 The platform will literally choke on the recapulation jobs. You have to mechanically map out those 00:14:20.979 --> 00:14:25.939 roles, teams and ownership models and migrate the data with the correct owner from day one. 00:14:25.939 --> 00:14:31.939 Okay, so guarding the data from internal users clicking around a UI is one piece of the puzzle. 00:14:31.940 --> 00:14:36.820 The security gauntlet handles that. But what happens when an automated external system, 00:14:36.820 --> 00:14:42.820 like a legacy enterprise platform or a data warehousing tool, tries to pull millions of rows 00:14:42.820 --> 00:14:48.100 out of data verse programmatically. That brings us back up to the API and integrations layer. 00:14:48.100 --> 00:14:53.700 Okay. The primary interfaces developers use to talk to the system are the Dataverse Web API, 00:14:53.700 --> 00:15:00.980 which implements the OData version four standard, the SDK for .NET and the TDS endpoint. Right, TDS. 00:15:00.980 --> 00:15:07.300 That stands for tabular data stream, which basically provides read only SQL style query access 00:15:07.300 --> 00:15:11.460 for reporting tool. Exactly. And the crucial rule here, the one that architects seem to get 00:15:11.460 --> 00:15:16.020 wrong time and time again, is keeping analytical workloads completely out of the operational 00:15:16.020 --> 00:15:21.220 transaction path. Absolutely vital. Dataverse is an operational platform. Its primary job is to 00:15:21.220 --> 00:15:26.900 run your live business transactions smoothly. Right. It's not a warehouse. No. If you have an 00:15:26.900 --> 00:15:31.380 external data science team that wants to run massive historical aggregation queries, 00:15:31.380 --> 00:15:37.140 like summing up five years of sales data, and they point their BI tools directly at the Dataverse 00:15:37.140 --> 00:15:43.220 Web API, they're going to consume all the computing resources that your live users need to do their 00:15:43.220 --> 00:15:48.900 jobs, which brings the whole system to a crawl. Dataverse data should flow to Microsoft fabric or 00:15:48.900 --> 00:15:54.100 one lake for analytics. Yeah. You do your heavy data warehousing there safely away from the APIs 00:15:54.180 --> 00:15:59.380 that your live apps rely on. Because if you overwhelm those live APIs with heavy queries, 00:15:59.380 --> 00:16:04.340 the system is going to protect itself. Yeah. Dataverse has built in service protection limits. 00:16:04.340 --> 00:16:08.500 Yes, it will push back. If an external system hammers the API too hard, 00:16:08.500 --> 00:16:14.820 Dataverse throws an HTTP 429 error, which explicitly includes a retry after signal. And here's where 00:16:14.820 --> 00:16:18.980 it gets really interesting. I've heard it's fascinating. I've seen integrations where developers treat 00:16:18.980 --> 00:16:24.020 a 429 error like a standard network blip. So you just program their system to aggressively 00:16:24.020 --> 00:16:28.900 retry the request immediately, which is the worst thing you can do. Wait, but from a developer's 00:16:28.900 --> 00:16:35.300 perspective, if a server drops my request, my first instinct is to try again immediately. So my user 00:16:35.300 --> 00:16:42.100 doesn't get a failure message. Why does Dataverse punish that? Because a 429 isn't a network drop. 00:16:42.100 --> 00:16:48.340 It is a deliberate protective action. Dataverse is explicitly telling you my CPU is overwhelmed, 00:16:48.340 --> 00:16:53.220 stop talking to me for exactly this many seconds. Oh, so it's a command, not a glitch. 00:16:53.220 --> 00:16:58.580 Exactly. If you ignore that mathematical pause and hit the API again instantly, 00:16:58.580 --> 00:17:03.620 the system registers you as a threat to its stability. It penalizes you further, 00:17:03.620 --> 00:17:07.780 pushing you deeper into the penalty box, and your application crashes harder. 00:17:07.780 --> 00:17:11.220 So you end up taking your own integration offline just by being impatient. 00:17:11.220 --> 00:17:16.340 Precisely why robust integrations require specific mechanical designs. You need back 00:17:16.339 --> 00:17:21.300 off logic, which means waiting exponentially longer between retries. You need bounded concurrency, 00:17:21.300 --> 00:17:25.939 which means strictly limiting how many active threads your external app opens against Dataverse 00:17:25.939 --> 00:17:31.220 at once. And most importantly, you need idempotency. Let's define idempotency for a second, 00:17:31.220 --> 00:17:35.060 because that's a turn that gets thrown around a lot mechanically. What does that actually look like? 00:17:35.060 --> 00:17:41.220 Idempotency means designing your request so that if a network drop does cause your external app to 00:17:41.220 --> 00:17:47.540 retry sending a create account payload three times, Dataverse knows to only create that account 00:17:47.540 --> 00:17:53.380 once. Rather than generating a mess of duplicate records. Exactly. You usually achieve this by 00:17:53.380 --> 00:17:58.820 passing a unique transaction ID with the payload. So Dataverse recognizes the duplicate request 00:17:58.820 --> 00:18:04.260 and safely ignores it. You have to design external systems to cooperate with Dataverse's limits, 00:18:04.260 --> 00:18:09.140 not fight them. And that philosophy extends to how we build internal business logic too. 00:18:09.140 --> 00:18:13.700 The source material strongly warns against turning every single business requirement 00:18:13.700 --> 00:18:19.220 into a synchronous plugin. Yes, a synchronous plugin operates on the main thread of the transaction. 00:18:19.220 --> 00:18:23.860 If a user clicks save and you have custom C sharp code running synchronously, 00:18:23.860 --> 00:18:28.100 the database transaction literally halts until your code finishes executing, 00:18:28.100 --> 00:18:31.940 which means if your code is reaching out to an external web service that takes five seconds to 00:18:31.940 --> 00:18:36.820 respond, the user is stuck staring at a loading spinner for five seconds. And doing that across 00:18:36.899 --> 00:18:41.620 thousands of users creates massive performance bottlenecks. Yeah, that scales terribly. 00:18:41.620 --> 00:18:45.780 It does. You have to ask yourself mechanically. Does this logic actually need to happen right 00:18:45.780 --> 00:18:51.139 this exact second to maintain data integrity? Or can it be handled by an asynchronous web hook 00:18:51.139 --> 00:18:55.939 or a service bus event in the background? It's all about respecting the platform's mechanical 00:18:55.939 --> 00:19:01.620 boundaries. But if an organization ignores all of this, if they ignore the metadata contract, 00:19:01.620 --> 00:19:06.659 if they dump everything into one storage type, if they let analytical queries hammer the live 00:19:06.659 --> 00:19:13.219 API's, they end up creating an incredibly fragile system. The total mess. Yeah. So how do support 00:19:13.219 --> 00:19:18.580 engineers prevent this chaos over the lifespan of the platform? Well, that brings us to governance 00:19:18.580 --> 00:19:24.020 and application lifecycle management or ALM, right? Data versus deeply integrated with power platform 00:19:24.020 --> 00:19:29.379 ALM, which uses the concept of solutions to package up your custom tables, security roles, 00:19:29.380 --> 00:19:34.980 flows and plugins into a single, deplorable file. Okay, solutions. The golden rule of ALM here 00:19:34.980 --> 00:19:39.860 is the strict flow of code from an unmanaged development environment to a managed testing 00:19:39.860 --> 00:19:44.260 environment. And finally, to a managed production environment. I've been on the other side of this 00:19:44.260 --> 00:19:49.940 though. You get a frantic call from a department head that a crucial form is broken in production. 00:19:49.940 --> 00:19:55.460 Oh boy. The fastest way to fix it is just to log in as an admin and tweak the form right there 00:19:55.540 --> 00:19:59.540 in the live environment. Yeah. But the source material treats this like a cardinal sin. 00:20:00.820 --> 00:20:05.940 Mechanically, it actually breaks when I do that. When you make an unmanaged change directly in 00:20:05.940 --> 00:20:12.980 production, you are creating an active unmanaged solution layer that sits on top of your managed 00:20:12.980 --> 00:20:18.420 code. Okay. Think of it like stacking transparencies on an overhead projector. Because of how the 00:20:18.420 --> 00:20:24.340 platform resolves conflicts, unmanaged layers always win. They always sit on top. I see where this is 00:20:24.340 --> 00:20:30.260 going. Yeah. So the next time your engineering team pushes a legitimate tested managed update 00:20:30.260 --> 00:20:34.899 from the development environment to fix that form permanently, your unmanaged quick fix is going to 00:20:34.899 --> 00:20:40.179 block it. It silently overrides the deployment pipeline. So the update says it deployed successfully, 00:20:40.179 --> 00:20:44.340 but the users don't see the changes because my late night hotfix is still sitting on top 00:20:44.340 --> 00:20:50.020 overriding the new code. Exactly. It is the single fastest way to destroy the integrity of your 00:20:50.020 --> 00:20:55.620 platform. That is terrifying. Production must remain strictly managed, meaning no direct 00:20:55.620 --> 00:21:00.980 customizations are ever made outside the pipeline. So what does this all mean for the boots on the 00:21:00.980 --> 00:21:07.780 ground support engineer who gets paged at 2 a.m. Because a legacy dataverse setup is falling apart. 00:21:07.780 --> 00:21:13.139 Well, the source outlines some specific enterprise anti patterns to look out for. Things like running 00:21:13.139 --> 00:21:18.980 broad select queries over the TDS endpoint, which joke the memory, right, or broad blanket auditing 00:21:18.980 --> 00:21:24.660 that fills up log capacity. High frequency polling from external apps that trigger 429 errors, 00:21:24.660 --> 00:21:30.260 or using dataverse as a telemetry warehouse. This raises an important question. What do all of 00:21:30.260 --> 00:21:36.020 those distinct anti patterns have in common? What's the thread? The root cause is always the exact 00:21:36.020 --> 00:21:41.460 same conceptual failure. It is an organization treating dataverse as an isolated database, 00:21:41.460 --> 00:21:46.420 instead of a governed application platform. Right back to where we started. Exactly. When you 00:21:46.420 --> 00:21:51.940 pull an API every two seconds, you are treating it like a local SQL server sitting under your desk. 00:21:51.940 --> 00:21:57.620 When you dump raw telemetry logs into it, you are completely ignoring the computational cost of 00:21:57.620 --> 00:22:02.900 the metadata and security layers that have to wrap every single row you insert. It's like trying to 00:22:02.900 --> 00:22:07.700 use a luxury sports car to tow a freight train. The engine isn't the problem. You're just applying 00:22:07.700 --> 00:22:12.340 the wrong machine to the workload. And eventually the transmission is going to blow. That's a perfect 00:22:12.419 --> 00:22:17.540 analogy. You have to engineer for the complex platform you actually have, not the simple database, 00:22:17.540 --> 00:22:21.220 you assume it is. And that architectural discipline is only going to become more critical as we move 00:22:21.220 --> 00:22:27.139 forward, right? Oh, absolutely. If we look at the source materials notes on the 2026 AI roadmap, 00:22:27.139 --> 00:22:32.740 Microsoft is heavily positioning gate averse as an AI grounded operational data platform. 00:22:32.740 --> 00:22:38.179 Wow. We are talking about features like work IQ, complex agent program ability, 00:22:38.180 --> 00:22:43.940 and deep native integration with Microsoft 365 co pilot, which means AI agents are essentially 00:22:43.940 --> 00:22:49.300 just the newest, most demanding consumers of the data versus API. They are an AI acts as a 00:22:49.300 --> 00:22:54.980 massive operational magnifier. How so? If your metadata is perfectly clean, your relational 00:22:54.980 --> 00:23:00.580 architecture is well defined. And your seven layer security gauntlet is properly constructed. 00:23:00.580 --> 00:23:05.540 An AI agent can do incredible things safely because it understands the platform contract. 00:23:05.620 --> 00:23:10.420 But if your metadata is messy, your security layers are flat and retrofitted and your 00:23:10.420 --> 00:23:15.780 integrations are brittle. AI will only amplify those underlying problems at light speed. 00:23:16.340 --> 00:23:20.500 Exactly. It will confidently surface sensitive data to users who shouldn't see it, 00:23:20.500 --> 00:23:24.980 or it will hallucinate business processes based on broken table relationships. 00:23:24.980 --> 00:23:29.940 Good AI requires a perfectly engineered data platform underneath it. It truly all comes back 00:23:29.940 --> 00:23:34.660 to the foundation security correctness, lifecycle repeatability through a lm, 00:23:34.660 --> 00:23:38.900 capacity predictability across those three buckets, and integration resilience against 00:23:38.900 --> 00:23:44.180 429 limits. It's all connected. Understanding those mechanisms is the difference between simply 00:23:44.180 --> 00:23:49.380 using data verse and truly engineering a resilient data verse platform. Before we wrap up, I want 00:23:49.380 --> 00:23:54.740 to leave you with one final thought to mow over based on that roadmap. If an autonomous AI agent 00:23:54.740 --> 00:23:59.540 running in co pilot tries to execute a business process on behalf of a user, it has to mechanically 00:23:59.540 --> 00:24:03.300 pass through that exact same seven layer security gauntlet we discussed earlier. 00:24:03.300 --> 00:24:08.659 So ask yourself, if an AI agent rapidly acted on your current data verse security model today, 00:24:08.659 --> 00:24:13.620 would it fail safely? Or would it expose things it shouldn't? Because the agents are already here 00:24:13.620 --> 00:24:17.139 and they're reading your platform contract right now. You've been listening to support 00:24:17.139 --> 00:24:21.300 engineering weekly from the Support Engineering Blog. For the full articles, references, and 00:24:21.300 --> 00:24:27.139 technical resources from today's discussion, visit the URL blog.sadhan.ch. Until next week, 00:24:27.140 --> 00:24:32.340 keep troubleshooting, keep learning, and keep engineering. Thank you for listening to support 00:24:32.340 --> 00:24:40.020 engineering weekly. This podcast is available on Apple podcast, Spotify, iHeartRadio, and other 00:24:40.020 --> 00:24:53.700 major podcast platforms. Follow the show on your preferred platform and never miss an episode.