WEBVTT 00:00:00.000 --> 00:00:03.600 You are listening to Support Engineering Weekly. 00:00:05.040 --> 00:00:05.760 You know the feeling. 00:00:06.000 --> 00:00:09.280 It's, um, it's honestly incredibly satisfying. 00:00:09.280 --> 00:00:13.000 You just drag and drop a trigger onto the canvas, add a few actions, 00:00:13.240 --> 00:00:14.600 authenticate a connector. 00:00:14.800 --> 00:00:18.000 And within minutes, you have a working power, automate flow. 00:00:18.080 --> 00:00:18.680 Oh, totally. 00:00:18.680 --> 00:00:20.520 I mean, it really feels like magic, right? 00:00:20.520 --> 00:00:23.360 The barrier to entry is just beautifully low. 00:00:23.360 --> 00:00:24.080 Yeah, exactly. 00:00:24.080 --> 00:00:29.280 You can automate some tedious daily tasks, like, I don't know, moving email 00:00:29.280 --> 00:00:33.040 attachments to a folder before your morning coffee even gets cold. 00:00:33.040 --> 00:00:33.400 Right. 00:00:33.400 --> 00:00:35.880 But then, you know, something happens. 00:00:36.240 --> 00:00:37.920 That simple little flow gets shared. 00:00:37.960 --> 00:00:39.160 Yeah, always. 00:00:39.200 --> 00:00:40.600 Another department starts using it. 00:00:40.600 --> 00:00:44.520 And suddenly your cute little five recorded a convenient script is 00:00:44.520 --> 00:00:46.600 processing thousands of transactions. 00:00:47.160 --> 00:00:50.480 It has quietly become, well, business critical software. 00:00:50.480 --> 00:00:50.680 Yep. 00:00:50.680 --> 00:00:53.920 And when it breaks, it is no longer just a minor annoyance. 00:00:53.920 --> 00:00:54.480 Oh no. 00:00:54.720 --> 00:00:56.400 It is a full blown production incident. 00:00:56.600 --> 00:00:57.480 I'm Alex, by the way. 00:00:57.480 --> 00:00:58.720 And thank you so much for joining us. 00:00:58.800 --> 00:00:59.400 And I'm Maya. 00:00:59.400 --> 00:01:00.519 We're really glad to have you here. 00:01:00.560 --> 00:01:03.000 Welcome to this deep dive for sport engineering weekly. 00:01:03.320 --> 00:01:08.160 Today, we have a very specific mission based on some brilliant engineering 00:01:08.160 --> 00:01:09.640 analysis by Satan Chandra. 00:01:09.720 --> 00:01:10.240 We do. 00:01:10.240 --> 00:01:14.280 We are going to fundamentally transform how you think about this platform. 00:01:14.320 --> 00:01:14.640 Right. 00:01:14.960 --> 00:01:19.880 We really need to shift from treating power, automate as just a simple workflow 00:01:19.880 --> 00:01:20.360 designer. 00:01:20.720 --> 00:01:25.840 We need to start engineering it as a robust, resilient production automation 00:01:25.840 --> 00:01:26.360 platform. 00:01:26.440 --> 00:01:30.120 And to make that transition, I mean, you have to realize that a flow in a 00:01:30.120 --> 00:01:34.280 production environment is never just a sequence of steps running in isolation. 00:01:34.320 --> 00:01:34.480 Right. 00:01:34.480 --> 00:01:35.360 It's much bigger than that. 00:01:35.400 --> 00:01:35.960 Exactly. 00:01:36.400 --> 00:01:40.240 Architecturally speaking, it is a small distributed system. 00:01:40.520 --> 00:01:44.520 So it carries all the risks, the latency issues and the failure 00:01:44.520 --> 00:01:46.520 requirements of any distributed architecture. 00:01:46.600 --> 00:01:50.200 Let's break down how we should actually be visualizing that architecture, because 00:01:50.200 --> 00:01:55.000 honestly viewing it as a vertical list of boxes on a screen, it kind of hides 00:01:55.000 --> 00:01:56.240 the actual complexity. 00:01:56.360 --> 00:01:57.520 It really does hide it. 00:01:57.800 --> 00:02:00.800 If you look at the architecture of true production automation, you need to 00:02:00.800 --> 00:02:03.200 visualize it across three distinct planes. 00:02:03.200 --> 00:02:03.480 Okay. 00:02:03.480 --> 00:02:03.960 Three planes. 00:02:03.960 --> 00:02:04.520 What's the first one? 00:02:04.720 --> 00:02:06.560 First, you have the control plane. 00:02:06.600 --> 00:02:08.160 This sits above your flow. 00:02:08.440 --> 00:02:13.360 It encompasses your governance, your application, lifecycle management, your 00:02:13.360 --> 00:02:16.960 data loss prevention policies and, you know, your environment strategy. 00:02:16.960 --> 00:02:17.120 Right. 00:02:17.120 --> 00:02:18.160 So the high level rules. 00:02:18.200 --> 00:02:18.360 Yeah. 00:02:18.760 --> 00:02:20.880 Then second, you have the execution plane. 00:02:21.120 --> 00:02:25.440 This is the power automate runtime itself, where your triggers, your actions 00:02:25.440 --> 00:02:27.720 and your concurrency are actually evaluated. 00:02:28.240 --> 00:02:28.560 Okay. 00:02:28.720 --> 00:02:32.800 And the third, finally sitting underneath all of that, you have the visibility 00:02:32.800 --> 00:02:33.120 plane. 00:02:33.120 --> 00:02:37.760 So this is your observability, your telemetry alerts, audit logs, all of that. 00:02:38.040 --> 00:02:38.800 That makes a lot of sense. 00:02:38.800 --> 00:02:43.480 So building a basic flow in the designers, really just building a single car. 00:02:43.920 --> 00:02:46.800 You put the engine in, attach the wheels and it drives down the driveway. 00:02:46.920 --> 00:02:47.520 Cause that's right. 00:02:47.520 --> 00:02:48.200 Exactly. 00:02:48.200 --> 00:02:48.960 You built a car. 00:02:49.079 --> 00:02:53.120 But building production automation is, well, it's like designing the entire 00:02:53.120 --> 00:02:53.880 highway system. 00:02:54.280 --> 00:02:55.880 You can't just put cars on the road. 00:02:55.920 --> 00:02:57.400 You have to account for traffic jams. 00:02:57.400 --> 00:02:58.079 Oh, absolutely. 00:02:58.160 --> 00:03:03.320 You need speed limits to prevent chaos, emergency lanes for when things break down 00:03:03.519 --> 00:03:05.760 and traffic cameras to see what's happening. 00:03:06.720 --> 00:03:11.040 The surrounding planes like governance and observability, that's the infrastructure 00:03:11.040 --> 00:03:12.359 making the highway viable. 00:03:12.480 --> 00:03:16.840 I love that analogy because without that infrastructure, a high speed pileup is 00:03:16.840 --> 00:03:18.160 basically inevitable. 00:03:18.200 --> 00:03:18.560 Right. 00:03:18.720 --> 00:03:23.400 When you deploy to production, that execution plane has to account for harsh 00:03:23.400 --> 00:03:24.120 realities. 00:03:24.480 --> 00:03:27.480 Volume dictates how much data is moving through. 00:03:27.880 --> 00:03:30.400 Concurrency dictates how many things happen at once. 00:03:30.400 --> 00:03:30.520 Yeah. 00:03:30.520 --> 00:03:34.080 And you have those constantly shifting dependencies on external APIs. 00:03:34.080 --> 00:03:34.320 Yeah. 00:03:34.400 --> 00:03:37.040 You always have to deal with APIs, which means, you know, the biggest 00:03:37.040 --> 00:03:39.360 danger isn't necessarily a flaw in the road itself. 00:03:39.360 --> 00:03:41.920 The biggest danger is how much traffic you dump onto it. 00:03:42.000 --> 00:03:42.600 Yes. 00:03:42.960 --> 00:03:46.920 And that brings us to performance engineering, specifically what is referred 00:03:46.920 --> 00:03:48.880 to as the multiplication problem. 00:03:48.880 --> 00:03:49.680 Well, this is a big one. 00:03:49.720 --> 00:03:53.280 It is because there seems to be a fundamental disconnect 00:03:53.280 --> 00:03:56.680 in how people visualize the work a flow is actually doing. 00:03:57.120 --> 00:03:57.960 Break that down for us. 00:03:57.960 --> 00:03:58.320 What do you mean? 00:03:58.680 --> 00:04:02.440 Well, the number of actions you see on your screen in the flow designer does not 00:04:02.440 --> 00:04:04.520 equal the workload generated by the flow. 00:04:04.800 --> 00:04:06.680 It almost always starts at the trigger. 00:04:06.720 --> 00:04:07.000 Right. 00:04:07.040 --> 00:04:07.920 The very first step. 00:04:08.040 --> 00:04:08.400 Yeah. 00:04:09.080 --> 00:04:13.360 The most massive anti-pattern we see is starting with a broad trigger, pulling 00:04:13.360 --> 00:04:18.120 in every possible record from a system and then using a condition action inside 00:04:18.120 --> 00:04:20.519 the flow to filter out what you don't need. 00:04:20.680 --> 00:04:25.919 So wait, you're triggering the flow 10,000 times just to say, never mind, to 9,900 of them. 00:04:25.960 --> 00:04:26.719 Exactly. 00:04:26.840 --> 00:04:31.960 And you pay a massive compute and platform limit penalty for work you didn't even need to do. 00:04:32.240 --> 00:04:32.760 Wow. 00:04:33.479 --> 00:04:35.000 That is hugely inefficient. 00:04:35.000 --> 00:04:35.719 It really is. 00:04:35.960 --> 00:04:38.240 The correct approach is to filter at the source. 00:04:38.520 --> 00:04:41.840 The pattern should always be a relevant event triggers the flow. 00:04:42.080 --> 00:04:45.919 You retrieve only what is strictly needed using scoped O data queries or trigger 00:04:45.919 --> 00:04:47.680 conditions, and then you process it. 00:04:47.680 --> 00:04:52.320 So if the API can do the filtering before the data ever reaches power automate, you let 00:04:52.320 --> 00:04:55.880 the API do the heavy lifting spot on, let the API do the work. 00:04:55.920 --> 00:05:00.680 And this multiplication problem gets exponentially wilder when we introduce loops, doesn't it? 00:05:00.960 --> 00:05:02.520 Oh, it gets terrifying. 00:05:02.960 --> 00:05:06.200 Let's say you have an apply to each loop inside that loop. 00:05:06.200 --> 00:05:08.320 You only have three simple actions. 00:05:08.360 --> 00:05:08.600 Right. 00:05:08.600 --> 00:05:10.000 So it looks tiny on the screen. 00:05:10.160 --> 00:05:11.280 It looks harmless. 00:05:11.520 --> 00:05:16.560 But if your trigger pulls in 2000 records and passes them into that loop, you didn't just 00:05:16.560 --> 00:05:17.960 build a three action flow. 00:05:18.160 --> 00:05:22.040 You generated 6000 individual connector requests instantly. 00:05:22.040 --> 00:05:24.240 Six thousand just from three little boxes. 00:05:24.280 --> 00:05:24.480 Yep. 00:05:24.880 --> 00:05:30.440 And every single one of those requests consumes API calls, memory and processing time. 00:05:31.000 --> 00:05:35.000 OK, here is where I want to push back a little bit on how we handle that volume. 00:05:35.000 --> 00:05:35.320 Sure. 00:05:35.360 --> 00:05:35.880 Go for it. 00:05:36.240 --> 00:05:43.240 Because if I am staring at a massive queue of thousands of records, my instinct as a 00:05:43.240 --> 00:05:45.680 problem solver is efficiency, right? 00:05:45.680 --> 00:05:50.240 Naturally, if I leave the concurrency control off, it processes one at a time. 00:05:50.440 --> 00:05:52.640 It might take hours or even hit a timeout limit. 00:05:53.200 --> 00:05:57.720 So why shouldn't I just go into the settings, crank up the concurrency control to like 50 00:05:57.840 --> 00:05:59.360 and run all those loops in parallel? 00:05:59.560 --> 00:06:02.240 Let the flow blast through the queue as fast as possible. 00:06:02.320 --> 00:06:06.520 It sounds like a great idea, but faster execution is not always better execution 00:06:06.520 --> 00:06:07.720 and a distributed system. 00:06:07.760 --> 00:06:08.200 Why not? 00:06:08.440 --> 00:06:10.480 Because your flow does not exist in a vacuum. 00:06:11.120 --> 00:06:14.160 Every action inside that loop is calling an API. 00:06:14.840 --> 00:06:19.400 If you crank up the concurrency, yeah, you shorten the runtime of the flow itself, 00:06:19.600 --> 00:06:22.640 but you simultaneously overwhelm your downstream dependencies. 00:06:22.840 --> 00:06:25.880 Oh, right, because they're taking all those hits at once. 00:06:26.080 --> 00:06:26.560 Exactly. 00:06:26.760 --> 00:06:31.480 You are firing thousands of requests per second at a SharePoint list and SQL 00:06:31.480 --> 00:06:34.440 database or some third party sauce app. 00:06:34.640 --> 00:06:36.440 So you aren't fixing the bottleneck? 00:06:36.640 --> 00:06:37.600 No, you're just moving it. 00:06:37.800 --> 00:06:40.680 And usually you're breaking the downstream system in the process. 00:06:40.680 --> 00:06:43.880 The downstream system isn't just going to quietly accept being 00:06:44.080 --> 00:06:46.480 d-dossed by our own internal automation, is it? 00:06:46.680 --> 00:06:48.120 No, it's going to defend itself. 00:06:48.320 --> 00:06:48.960 How does it do that? 00:06:49.160 --> 00:06:51.360 It will defend itself through throttling. 00:06:51.560 --> 00:06:56.400 When you overwhelm a dependency, you get slapped with an HTTP 429 error. 00:06:56.600 --> 00:06:57.800 Too many requests. 00:06:58.000 --> 00:06:59.800 The dreaded 429. 00:07:00.000 --> 00:07:04.760 Yep. Or if you are hitting data verse, you hit service protection limits. 00:07:04.960 --> 00:07:09.160 The API essentially tells your flow to back off because it simply cannot process 00:07:09.160 --> 00:07:09.800 any more data. 00:07:09.880 --> 00:07:15.000 So a resilient production flow needs a built in strategy for those 429 errors. 00:07:15.200 --> 00:07:16.600 It can't just fail the whole run. 00:07:16.800 --> 00:07:17.160 Right. 00:07:17.160 --> 00:07:19.760 It requires dynamic back off intervals. 00:07:19.960 --> 00:07:25.200 The flow needs to read the retry after header from the rejected request, 00:07:25.400 --> 00:07:28.480 wait that specific amount of time and then try again. 00:07:28.680 --> 00:07:29.320 That makes sense. 00:07:29.520 --> 00:07:33.640 But your retry strategy has to be a conscious part of your workload model. 00:07:33.840 --> 00:07:35.000 It can't be an afterthought. 00:07:35.200 --> 00:07:37.400 Think about buying concept tickets online. 00:07:37.560 --> 00:07:41.039 You know, the website freezes, you panic and you double click the checkout button. 00:07:41.239 --> 00:07:42.000 We've all been there. 00:07:42.200 --> 00:07:42.679 Right. 00:07:42.880 --> 00:07:45.679 If a flow sends a request to create a record, 00:07:45.880 --> 00:07:49.359 times out waiting for the response and then automatically retries, 00:07:49.560 --> 00:07:51.520 we risk creating a duplicate record. 00:07:51.719 --> 00:07:52.039 Exactly. 00:07:52.239 --> 00:07:55.520 We know we need a dampadency so we don't accidentally buy two tickets. 00:07:55.719 --> 00:08:00.080 But actually achieving a dampadency across different third party APIs and power 00:08:00.280 --> 00:08:02.599 automated is notoriously difficult. 00:08:02.799 --> 00:08:03.919 Yeah. How do we build that in? 00:08:04.120 --> 00:08:05.719 So retries are actually safe. 00:08:05.800 --> 00:08:09.840 You have to utilize stable business keys instead of relying on auto generated 00:08:10.040 --> 00:08:10.760 database IDs. 00:08:10.960 --> 00:08:12.120 OK, give me an example of that. 00:08:12.320 --> 00:08:14.240 Let's say you are sinking an invoice. 00:08:14.440 --> 00:08:16.640 You don't just send a create record command. 00:08:16.840 --> 00:08:20.760 You first query the system using the unique invoice number as a correlation 00:08:20.960 --> 00:08:25.280 ID and wherever possible you use up cert style operations. 00:08:25.480 --> 00:08:27.720 Up cert, meaning up data insert. 00:08:27.920 --> 00:08:28.440 Exactly. 00:08:28.640 --> 00:08:31.800 An up cert checks if the record exists based on that stable key. 00:08:32.000 --> 00:08:33.600 If it exists, it updates it. 00:08:33.800 --> 00:08:35.160 If it doesn't, it inserts it. 00:08:35.159 --> 00:08:35.799 Odd smart. 00:08:36.000 --> 00:08:36.400 Right. 00:08:36.600 --> 00:08:40.679 The operation is designed so that if the exact same business event is retried 00:08:40.880 --> 00:08:45.760 three times because of network latency, it yields the exact same intended outcome 00:08:45.959 --> 00:08:47.600 without unintended side effects. 00:08:47.800 --> 00:08:52.240 So if your flow crosses a system boundary, making retrial safe is absolutely 00:08:52.439 --> 00:08:57.480 non-negotiable, but what happens if a retry isn't safe or what if the back 00:08:57.680 --> 00:09:01.560 off maxes out after four attempts and the action just completely permanently fails, 00:09:01.760 --> 00:09:01.880 right? 00:09:02.079 --> 00:09:05.000 In a production system, it can't just freeze and hope someone notices. 00:09:05.160 --> 00:09:08.240 No, the system needs to know exactly where to route the failure. 00:09:08.439 --> 00:09:09.120 How do we do that? 00:09:09.319 --> 00:09:13.400 Error handling must be an intentional explicit part of the architecture. 00:09:13.600 --> 00:09:15.760 In a production flow, you use what are called 00:09:15.959 --> 00:09:19.959 scopes combined with run after conditions to build those routing paths. 00:09:20.160 --> 00:09:23.800 That sounds exactly like a try catch block in traditional software engineering. 00:09:24.000 --> 00:09:26.719 It is entirely the power automated equivalent. 00:09:26.920 --> 00:09:30.360 You put your main business logic inside a primary scope. 00:09:30.560 --> 00:09:31.520 That's your try block. 00:09:31.720 --> 00:09:35.000 OK. And if everything succeeds, the flow moves down the happy path. 00:09:35.160 --> 00:09:35.880 Exactly. 00:09:36.079 --> 00:09:40.560 But you create a second scope directly underneath it, configured to only run 00:09:40.760 --> 00:09:43.520 after the first scope fails or times out. 00:09:43.719 --> 00:09:44.959 That is your catch block. 00:09:45.160 --> 00:09:49.480 So the failure path takes over, but the failure path needs to be smart enough 00:09:49.680 --> 00:09:51.719 to know what kind of failure just happened. 00:09:51.920 --> 00:09:55.920 Yes, it has to differentiate because transient failures like throttling get 00:09:56.120 --> 00:09:58.040 those controlled retries we just discussed. 00:09:58.240 --> 00:10:01.439 But what if it's a business rule failure, like, I don't know, a missing 00:10:01.640 --> 00:10:02.319 approval manager? 00:10:02.320 --> 00:10:06.440 In that case, the flow needs to capture that and route it to a human queue for review. 00:10:06.640 --> 00:10:08.600 Or what about a configuration failure? 00:10:08.800 --> 00:10:12.440 Say a service count password expires and a connector gets revoked. 00:10:12.640 --> 00:10:15.680 Retrying a bad password 50 times isn't going to fix it. 00:10:15.880 --> 00:10:19.080 Definitely not. In that scenario, the failure path must immediately 00:10:19.280 --> 00:10:23.200 terminate the retry loop, stop the flow and escalate a critical alert to an 00:10:23.400 --> 00:10:24.120 operations team. 00:10:24.320 --> 00:10:28.320 But if our flow is escalating these critical failures, 00:10:28.520 --> 00:10:32.280 the operator on the other end is flying blind unless they have the tool 00:10:32.320 --> 00:10:34.240 of telemetry to see what just crashed. 00:10:34.440 --> 00:10:35.120 Very true. 00:10:35.320 --> 00:10:39.360 My instinct here is to say we should just log absolutely every single step, 00:10:39.560 --> 00:10:44.120 every variable and every JSON payload in the flow to a custom database. 00:10:44.320 --> 00:10:46.720 That way troubleshooting is incredibly easy. 00:10:46.920 --> 00:10:49.160 Why is that considered an anti-pattern? 00:10:49.360 --> 00:10:53.000 Because you will drown yourself in noise and you'll fit data storage limits 00:10:53.200 --> 00:10:54.040 incredibly fast. 00:10:54.240 --> 00:10:54.760 Oh, I see. 00:10:54.960 --> 00:10:57.960 The goal of observability is actionable evidence. 00:10:58.160 --> 00:11:01.920 If your operating model requires an engineer to manually open individual, 00:11:02.000 --> 00:11:05.880 graphical flow runs in the power automate UI to figure out what went wrong. 00:11:06.079 --> 00:11:08.160 Your model is not sustainable at scale. 00:11:08.360 --> 00:11:12.319 Yeah, clicking through a haystack of green check marks and red Xs for a flow 00:11:12.520 --> 00:11:15.079 that runs 10,000 times a day, that sounds impossible. 00:11:15.280 --> 00:11:15.920 It is impossible. 00:11:16.120 --> 00:11:19.640 So how do we get that actionable evidence without logging the entire 10 megabyte 00:11:19.839 --> 00:11:20.959 payload of a file transfer? 00:11:21.160 --> 00:11:24.959 A true production operating model uses application insights and KQL 00:11:25.160 --> 00:11:27.839 Cousteau query language for fleet level diagnostics. 00:11:28.040 --> 00:11:29.560 OK, so moving away from the UI. 00:11:29.719 --> 00:11:30.479 Exactly. 00:11:30.479 --> 00:11:34.880 Instead of clicking through a UI, an engineer writes a KQL query to instantly 00:11:35.079 --> 00:11:38.319 aggregate failure rates across thousands of runs. 00:11:38.520 --> 00:11:43.079 They can pinpoint a sudden spike in four hundred and twenty nine throttling errors 00:11:43.280 --> 00:11:45.520 across the whole environment in seconds. 00:11:45.719 --> 00:11:46.959 That is so much faster. 00:11:47.160 --> 00:11:47.359 Right. 00:11:47.560 --> 00:11:51.119 You only use custom logging for specific business context. 00:11:51.319 --> 00:11:55.520 You log the business key, the correlation ID and the specific error state. 00:11:55.719 --> 00:11:58.319 You leave the heavy payload in the source system. 00:11:58.480 --> 00:11:59.520 That makes perfect sense. 00:11:59.720 --> 00:11:59.920 Yeah. 00:12:00.120 --> 00:12:04.080 But if a system requires this level of custom telemetry, explicit failure 00:12:04.280 --> 00:12:07.880 routing and I dampen it in retries, it is highly engineered software. 00:12:08.080 --> 00:12:08.800 Yes, it is. 00:12:09.000 --> 00:12:11.840 You can't just leave that running under the user credentials of the person who 00:12:12.040 --> 00:12:13.880 originally clicked create flow, right? 00:12:14.080 --> 00:12:18.160 Tying a distributed architecture to a single employees user account feels like 00:12:18.360 --> 00:12:19.680 a massive operational risk. 00:12:19.880 --> 00:12:20.240 It is. 00:12:20.440 --> 00:12:23.280 The human element is the most dangerous dependency. 00:12:23.480 --> 00:12:27.200 If a business critical automation depends on a single employee's credentials, 00:12:27.360 --> 00:12:32.000 the workflow stops entirely the moment they change roles or leave the company or 00:12:32.200 --> 00:12:34.720 their password policies force a reset during a weekend. 00:12:34.920 --> 00:12:35.760 Exactly. 00:12:35.960 --> 00:12:39.960 For production flows, you must advocate for service principal ownership. 00:12:40.160 --> 00:12:45.520 You have to separate the identity running the flow from any actual human user, 00:12:45.720 --> 00:12:48.680 which ties directly into how we deploy the software. 00:12:48.880 --> 00:12:51.360 We use application lifecycle management or ALM. 00:12:51.560 --> 00:12:53.480 Yes, ALM is crucial. 00:12:53.680 --> 00:12:56.360 We know development happens in unmanaged solutions. 00:12:56.520 --> 00:13:01.560 But test and production environments should only receive managed solutions. 00:13:01.760 --> 00:13:06.200 Why do managed solutions specifically protect the production environment? 00:13:06.400 --> 00:13:09.840 Because a managed solution locks down the definition of the flow, 00:13:10.040 --> 00:13:13.320 it prevents users from making direct edits in the production environment. 00:13:13.520 --> 00:13:14.920 So no quick fixes on the fly. 00:13:15.120 --> 00:13:15.280 Right. 00:13:15.480 --> 00:13:18.080 You cannot just hit edit, change a variable and hit save. 00:13:18.280 --> 00:13:21.680 All changes must be made in the unmanaged development environment committed to 00:13:21.880 --> 00:13:24.800 version control and pushed through a deployment pipeline. 00:13:24.880 --> 00:13:28.719 It guarantees that the code running in production is the exact code that passed 00:13:28.920 --> 00:13:33.439 testing moving a managed solution from development to production introduces a 00:13:33.640 --> 00:13:35.040 mechanical challenge, though. 00:13:35.240 --> 00:13:38.719 The database connections or email addresses are going to change between those 00:13:38.920 --> 00:13:39.640 environments. 00:13:39.839 --> 00:13:41.920 How do we handle that without editing the flow? 00:13:42.120 --> 00:13:45.520 You keep configuration entirely out of the flow definition. 00:13:45.719 --> 00:13:46.520 Oh, interesting. 00:13:46.719 --> 00:13:48.920 How you use environment variables for your settings, 00:13:49.120 --> 00:13:53.280 like SharePoint site URLs or email addresses and connection references for your 00:13:53.479 --> 00:13:54.120 authentication. 00:13:54.200 --> 00:13:54.600 I see. 00:13:54.800 --> 00:13:57.880 So when the managed solution moves from dev to test to prod, 00:13:58.080 --> 00:14:02.440 the flow definition remains identical, but the environment variables dynamically 00:14:02.640 --> 00:14:05.640 swap in the correct production endpoints when the package arrives. 00:14:05.840 --> 00:14:07.440 That is incredibly clean. 00:14:07.640 --> 00:14:09.320 Oh, so we have it properly deployed. 00:14:09.520 --> 00:14:10.760 It's running under a service principle. 00:14:10.960 --> 00:14:11.720 It's fully managed. 00:14:11.920 --> 00:14:13.120 That's perfect so far. 00:14:13.320 --> 00:14:17.280 But over time, the business keeps asking for just one more feature. 00:14:17.480 --> 00:14:19.800 Oh, always scope creep. 00:14:20.000 --> 00:14:20.240 Right. 00:14:20.320 --> 00:14:24.760 They want the flow to send a message to a team channel and then update a 00:14:24.960 --> 00:14:27.720 spreadsheet and then generate a PDF report. 00:14:27.920 --> 00:14:31.200 Suddenly we have created the dreaded mega flow. 00:14:31.400 --> 00:14:31.960 A mega flow. 00:14:32.160 --> 00:14:32.320 Yes. 00:14:32.520 --> 00:14:36.440 How do we know when a flow has officially become a mega flow that needs to be broken 00:14:36.640 --> 00:14:36.720 apart? 00:14:36.920 --> 00:14:40.720 Well, the most glaring warning sign is unrelated responsibilities. 00:14:40.920 --> 00:14:45.720 If your flow is reading an inbox, parsing data, updating a database and handling 00:14:45.920 --> 00:14:49.040 employee onboarding all at once, you have built a monolith. 00:14:49.240 --> 00:14:49.480 Wow. 00:14:49.560 --> 00:14:50.920 Yeah, that is way too much. 00:14:51.120 --> 00:14:55.880 Another sign is deep impossible to read nesting of conditional statements where you 00:14:56.080 --> 00:14:58.640 are eight levels deep in if yes, branches. 00:14:58.840 --> 00:15:01.720 Do those mega flows fail differently than modular ones? 00:15:01.920 --> 00:15:02.360 They do. 00:15:02.560 --> 00:15:07.720 And that reveals the most critical red flag unrelated failure domains. 00:15:07.920 --> 00:15:09.760 Think about the execution boundaries. 00:15:09.960 --> 00:15:10.200 OK. 00:15:10.400 --> 00:15:14.440 If the step that sends a notification email fails because of a temporary network 00:15:14.640 --> 00:15:19.040 blip, should it cause the entire database update process to fail and roll back? 00:15:19.199 --> 00:15:20.240 No, obviously not. 00:15:20.439 --> 00:15:20.599 Right. 00:15:20.800 --> 00:15:24.560 So if the answer is no, those two tasks should not be tightly coupled in the same 00:15:24.760 --> 00:15:25.360 execution path. 00:15:25.560 --> 00:15:25.839 Got it. 00:15:26.040 --> 00:15:29.439 So you split the flow to create clear execution and failure boundaries. 00:15:29.640 --> 00:15:33.520 One parent flow can receive the workload and it can hand off the heavy data 00:15:33.719 --> 00:15:37.839 processing to a child flow and hand off the notification to another child flow. 00:15:38.040 --> 00:15:40.280 Yes, they execute independently. 00:15:40.480 --> 00:15:43.000 Modularity isolates your failures. 00:15:43.199 --> 00:15:48.240 The parent flow acts as an orchestrator while the child flows act as specialized 00:15:48.320 --> 00:15:48.920 workers. 00:15:49.120 --> 00:15:49.600 Exactly. 00:15:49.800 --> 00:15:53.920 But splitting a mega flow apart often reveals something a bit uncomfortable. 00:15:54.120 --> 00:15:57.360 Some of these tasks probably shouldn't be handled by power automated at all. 00:15:57.560 --> 00:15:58.000 Oh, for sure. 00:15:58.200 --> 00:16:02.320 I've seen cloud flows timeout when they try to process a massive data set or run 00:16:02.520 --> 00:16:03.720 complex mathematical loops. 00:16:03.920 --> 00:16:06.760 It feels like trying to run a marathon in a sprint lane. 00:16:06.960 --> 00:16:11.560 If power automate can't handle the heavy compute, where do we upload that? 00:16:11.760 --> 00:16:15.040 That is the ultimate architectural maturity moment. 00:16:15.199 --> 00:16:19.760 The goal of enterprise governance is not to maximize your usage of power automate. 00:16:19.959 --> 00:16:20.640 That's a good point. 00:16:20.839 --> 00:16:25.240 The goal is to build the most reliable automation system using the right tools. 00:16:25.439 --> 00:16:28.839 Cloud flows are orchestration engines, not compute engines. 00:16:29.040 --> 00:16:32.319 They're subject to strict timeout limits and API constraints. 00:16:32.520 --> 00:16:36.520 So what's the alternative for the heavy lifting for heavy complex compute? 00:16:36.719 --> 00:16:40.920 You need to hand that workload off to a dedicated service like Azure functions, 00:16:41.120 --> 00:16:44.760 which can process the code efficiently and return the result to the flow. 00:16:44.920 --> 00:16:50.360 OK, but what if the task involves interacting with a legacy mainframe or say 00:16:50.560 --> 00:16:53.360 a web portal that doesn't have an API available? 00:16:53.560 --> 00:16:57.800 A cloud flow can't send an HTTP request if there's no endpoint to receive it. 00:16:58.000 --> 00:17:03.040 That is when you pivot to desktop flows, which is robotic process automation or RPA. 00:17:03.240 --> 00:17:03.840 RPA, right? 00:17:04.040 --> 00:17:07.280 RPA simulates human interaction with the user interface. 00:17:07.480 --> 00:17:12.119 It literally clicks the buttons and reads the screen when modern API simply don't 00:17:12.319 --> 00:17:12.680 exist. 00:17:12.760 --> 00:17:15.800 And what if the workload is incredibly spiky? 00:17:16.000 --> 00:17:20.280 Let's say we get zero traffic all week and then 10,000 requests hit the system 00:17:20.480 --> 00:17:23.240 in five minutes on a Friday afternoon, a classic scenario. 00:17:23.440 --> 00:17:26.440 If we dumped 10,000 concurrent triggers directly into a flow, 00:17:26.640 --> 00:17:30.320 it's going to hit platform limits, start returning four twenty nines to the source 00:17:30.520 --> 00:17:31.920 and we will permanently drop data. 00:17:32.120 --> 00:17:36.680 Yes, which is why for spiky workloads, you never expose the flow directly to the source 00:17:36.880 --> 00:17:37.760 event. What do you do instead? 00:17:37.960 --> 00:17:41.040 You put a message queue in front of the process like Azure service bus. 00:17:41.280 --> 00:17:43.159 The message QAC is a shock absorber. 00:17:43.360 --> 00:17:46.840 It durably accepts and holds all 10,000 requests instantly. 00:17:47.040 --> 00:17:47.440 Oh, I see. 00:17:47.639 --> 00:17:52.080 Then your flow is configured to pull messages off that queue at a safe controlled 00:17:52.280 --> 00:17:53.120 concurrency rate. 00:17:53.320 --> 00:17:57.879 The queue absorbs the spike and the flow processes the backlog steadily without 00:17:58.080 --> 00:17:58.920 ever hitting a limit. 00:17:59.120 --> 00:18:00.879 That makes perfect architectural sense. 00:18:01.080 --> 00:18:04.280 It is all incredibly structured deterministic logic. 00:18:04.480 --> 00:18:08.320 But you know, looking at the broader landscape of technology right now, I have 00:18:08.320 --> 00:18:10.520 to ask about the impact of artificial intelligence. 00:18:10.720 --> 00:18:12.720 AI, I knew we'd get here. 00:18:12.919 --> 00:18:16.919 We have to with AI agents becoming capable of independent reasoning, 00:18:17.120 --> 00:18:18.760 writing code and navigating systems. 00:18:18.960 --> 00:18:22.679 Are these rigid, carefully engineered workflows becoming obsolete? 00:18:22.879 --> 00:18:27.919 Should we stop building complex try catch scopes and just hand the goal to an AI 00:18:28.120 --> 00:18:29.679 agent to figure out the routing on the fly? 00:18:29.879 --> 00:18:32.639 It is the most common architectural debate happening right now. 00:18:32.840 --> 00:18:36.399 But assuming AI replaces structured automation relies on a misunderstanding 00:18:36.600 --> 00:18:38.120 of what workflows actually do. 00:18:38.480 --> 00:18:39.200 Really? How so? 00:18:39.399 --> 00:18:43.439 We have to clarify the distinction between a deterministic workflow and an 00:18:43.639 --> 00:18:44.480 agent workflow. 00:18:44.679 --> 00:18:46.280 Where is the line between them? 00:18:46.480 --> 00:18:50.000 Well, a deterministic workflow, what we've been talking about this whole time, 00:18:50.200 --> 00:18:55.600 is used for processes that require strict, predictable and audible rules. 00:18:55.800 --> 00:19:00.679 Think of financial compliance, updating a system of record or sinking inventory. 00:19:00.879 --> 00:19:03.679 Right. You absolutely do not want creativity in those processes. 00:19:03.879 --> 00:19:08.000 No, you want the system to do exactly what it is told, the exact same way, 00:19:08.080 --> 00:19:11.760 every single time producing a predictable JSON schema. 00:19:11.960 --> 00:19:15.560 Yeah, I definitely don't want a non deterministic, large language model getting 00:19:15.760 --> 00:19:20.320 creative with my payroll deductions or guessing how to update a database schema. 00:19:20.520 --> 00:19:21.280 Exactly. 00:19:21.480 --> 00:19:25.520 An agentic workflow, on the other hand, is utilized when reasoning and adaptation 00:19:25.720 --> 00:19:30.160 are genuinely needed because the exact path to the solution isn't known in advance. 00:19:30.360 --> 00:19:30.720 Like what? 00:19:30.920 --> 00:19:34.960 For example, analyzing a furious customer email, extracting the sentiment, 00:19:35.040 --> 00:19:38.800 determining which internal department is best equipped to handle it and drafting 00:19:39.000 --> 00:19:42.640 a customer response, that requires reasoning for they aren't competitors fighting 00:19:42.840 --> 00:19:43.840 for the same workloads. 00:19:44.040 --> 00:19:44.640 Not at all. 00:19:44.840 --> 00:19:46.840 They are completely complimentary. 00:19:47.040 --> 00:19:52.520 The deterministic workflow handles the rigid governed data movement when it reaches 00:19:52.720 --> 00:19:57.240 a step that requires human like reasoning, the workflow calls out to an AI agent. 00:19:57.440 --> 00:19:57.960 I see. 00:19:58.160 --> 00:20:02.640 It waits for the agent to make a decision and then takes that decision and executes 00:20:02.640 --> 00:20:06.720 it safely within the governed, auditable boundaries of the deterministic system. 00:20:06.920 --> 00:20:08.040 That is fascinating. 00:20:08.240 --> 00:20:12.280 Let's synthesize everything we've covered today because this really is a massive paradigm 00:20:12.480 --> 00:20:12.640 shift. 00:20:12.840 --> 00:20:13.480 Truly is. 00:20:13.680 --> 00:20:17.880 What we've learned is that a production flow is fundamentally a distributed system. 00:20:18.080 --> 00:20:22.160 It requires intentional workload management, filtering at the source, controlling 00:20:22.360 --> 00:20:24.400 loops and respecting downstream limits. 00:20:24.600 --> 00:20:24.680 Yes. 00:20:24.880 --> 00:20:25.680 And it demands safe, 00:20:25.880 --> 00:20:29.680 a depotent retries and explicit error paths using scopes. 00:20:29.880 --> 00:20:30.000 Right. 00:20:30.080 --> 00:20:35.160 It requires actionable observability using tools like KQL rather than just logging noise 00:20:35.360 --> 00:20:39.440 and it relies on durable application lifecycle management, service principles 00:20:39.640 --> 00:20:42.240 and the right architectural tools to survive in the real world. 00:20:42.440 --> 00:20:46.040 Applying those principles means you stop hoping the flow succeeds and you start 00:20:46.240 --> 00:20:50.120 actively engineering it to survive when things inevitably fail, which brings us to a 00:20:50.320 --> 00:20:53.920 final provocative thought for you to take away from this deep dive. 00:20:54.120 --> 00:20:57.520 I want you to think about the environment you manage right now. 00:20:57.680 --> 00:21:02.160 Look at the most successful flows in your organization, the ones that have been 00:21:02.360 --> 00:21:06.040 running quietly in the background for months, the ones nobody worries about. 00:21:06.240 --> 00:21:06.400 Yeah. 00:21:06.600 --> 00:21:07.520 Take a close look at those. 00:21:07.720 --> 00:21:12.480 Are they actually engineered for failure, observability and scale or have they just 00:21:12.680 --> 00:21:13.480 been lucky so far? 00:21:13.680 --> 00:21:17.440 Because in a distributed system, luck always runs out eventually. 00:21:17.639 --> 00:21:21.280 You've been listening to Support Engineering Weekly from a Support Engineering 00:21:21.480 --> 00:21:25.200 Blog. For the full articles, references and technical resources from today's 00:21:25.240 --> 00:21:29.240 discussion, visit the URL blog.sadhan.ch. 00:21:29.440 --> 00:21:34.160 Until next week, keep troubleshooting, keep learning and keep engineering. 00:21:34.360 --> 00:21:36.040 Thank you for listening to Support 00:21:36.240 --> 00:21:37.560 Engineering Weekly. 00:21:37.760 --> 00:21:43.760 This podcast is available on Apple podcast, Spotify, iHeartRadio and other 00:21:43.960 --> 00:21:46.000 major podcast platforms. 00:21:46.200 --> 00:21:51.520 Follow the show on your preferred platform and never miss an episode.