WEBVTT 00:00:00.000 --> 00:00:08.400 You are listening to Support Engineering Weekly. I'm Alex and I'm Maya. And we want to 00:00:08.400 --> 00:00:12.320 sincerely thank you for joining in this podcast today. You know, usually when we talk about 00:00:12.320 --> 00:00:16.960 enterprise infrastructure, there's this expectation of, well, precision. Right, like it should just 00:00:16.960 --> 00:00:21.680 work. Exactly. I mean, if you think about engineering a physical structure, like, I don't know, a 00:00:21.680 --> 00:00:27.040 suspension bridge, the parameters are completely tangible. A steel beam fracture is a stress 00:00:27.040 --> 00:00:31.439 sensor flashes red on a dashboard. And the lead engineer just points at the screen and says, 00:00:31.439 --> 00:00:37.920 you know, there it is. That's the problem. We swap that part out. Because it's entirely binary. 00:00:37.920 --> 00:00:43.440 I mean, the physical world gives us these measurable, very localized failures. Broken or not broken. 00:00:43.440 --> 00:00:48.080 It's clean. But then you step into the world of modern cloud architecture. And suddenly, 00:00:48.080 --> 00:00:53.440 that sensor dashboard is completely dark. We're looking at a diagnostic landscape. Yeah. That 00:00:53.440 --> 00:00:58.160 is honestly just terrifying. Oh, absolutely. It's a nightmare for the people on call. Right. 00:00:58.160 --> 00:01:04.080 Because how do you support an environment when the cloud's own telemetry goes blind? Or when 00:01:04.080 --> 00:01:09.520 you're cutting edge AI assistant suddenly forgets how to read last month's emails? Or the best one, 00:01:09.520 --> 00:01:15.840 when a routine update breaks copy and paste? Yes. That is my absolute favorite from this week's batch. 00:01:16.880 --> 00:01:22.720 When an invisible backend update somehow completely breaks the copy and paste function in a simple 00:01:22.720 --> 00:01:28.720 spreadsheet. That is the absolute definition of diagnostic muddy waters. The complexity of 00:01:28.720 --> 00:01:33.440 these systems has reached a point where, you know, the symptoms often look absolutely nothing like 00:01:33.440 --> 00:01:39.440 the root cause. Yeah. You pull a thread over here and a seemingly unrelated feature just unravels 00:01:39.440 --> 00:01:44.240 entirely across the globe, which is the core reason we're doing this deep dive. Today, 00:01:44.240 --> 00:01:50.000 our mission is to analyze the Microsoft service health updates from the week of September 14 to 18 00:01:50.080 --> 00:01:55.200 2026. We have a stack of incident reports that if you just glance at them, well, they look like 00:01:55.200 --> 00:01:59.600 isolated random glitches, but they really aren't. When you stack them together and examine the 00:01:59.600 --> 00:02:04.160 underlying mechanics, they tell this fascinating, highly interconnected story. It's all about 00:02:04.160 --> 00:02:10.000 architectural stress, hidden data dependencies, and honestly, the very real growing pains of 00:02:10.000 --> 00:02:16.080 integrating AI into legacy systems. We are looking at a perfect cross section of what it actually 00:02:16.080 --> 00:02:20.720 means to keep the lights on in a highly entangled modern tech stack. 00:02:20.720 --> 00:02:24.880 So let's start with the most complex new layer of that stack artificial intelligence. 00:02:24.880 --> 00:02:29.120 We had a remarkably rough week for Microsoft 365 co-pilot. That's for sure. 00:02:29.120 --> 00:02:36.240 We really did. We saw a broad outage incident CP 1470554 where essentially the entire co-pilot 00:02:36.240 --> 00:02:42.080 ecosystem just fell over. I mean, chat notebooks, word, excel, co-pilot studio. 00:02:42.160 --> 00:02:46.880 Just a total blackout. Yeah. And the root cause was listed super generically as a configuration 00:02:46.880 --> 00:02:51.440 change in request processing. But what really caught my eye was this other issue. Incident 00:02:51.440 --> 00:02:57.920 CW 1470765. Oh, the co-pilot chat retrieval failure. That one is a masterpiece of complex 00:02:57.920 --> 00:03:02.720 system failure. Okay. So let's unpack the mechanics of this. A misconfiguration in the semantic index 00:03:02.720 --> 00:03:07.840 model meant co-pilot just couldn't retrieve any exchange online emails dated after July 22, 00:03:08.160 --> 00:03:12.240 26th. It literally just forgot they existed. Right. It's like co-pilot suddenly lost its 00:03:12.240 --> 00:03:16.720 reading glasses for anything written the last two months. But here's the crazy part. 00:03:17.520 --> 00:03:22.400 If you prompted the chat with the exact word for word quote from the missing email, 00:03:22.960 --> 00:03:27.760 it would suddenly remember it and pull it up, which is so bizarre from a user perspective. 00:03:27.760 --> 00:03:33.840 Wait. So by yelling the exact quote, you were actually bypassing the broken AI brain and 00:03:33.840 --> 00:03:39.120 forcing it into some kind of legacy lookup mode. Exactly. You triggered its fail safe. 00:03:39.120 --> 00:03:44.160 It's jarring because users expect an intelligent assistant and suddenly they are interacting with 00:03:44.160 --> 00:03:49.440 this very literal rigid database query tool. Yeah. Like going back to the dark ages. 00:03:49.440 --> 00:03:54.560 Basically, what we're seeing here is the fundamental difference in how these two search paradigms 00:03:54.560 --> 00:03:59.360 actually process information. When we talk about traditional keyword indexing, it's rigid, 00:03:59.360 --> 00:04:04.400 but it's incredibly reliable. Like a massive lookup table. Yes. If you search for the word budget, 00:04:04.400 --> 00:04:10.160 the system scans the table for the exact string of characters, B-U-D-G-E-T, and returns the matches. 00:04:10.160 --> 00:04:15.120 The classic CTR-L approach. It either finds the letters in that exact order or it doesn't. 00:04:15.120 --> 00:04:22.000 Precisely. But semantic indexing, which is the core engine powering co-pilots contextual 00:04:22.000 --> 00:04:26.880 understanding, well, it operates on a completely different plane. It doesn't just look at strings 00:04:26.880 --> 00:04:32.880 of characters. It's doing something more conceptual. Right. It translates language into mathematical 00:04:32.880 --> 00:04:38.880 vectors, what we call embeddings. Imagine taking a sentence and turning it into coordinates on a 00:04:38.880 --> 00:04:44.800 multi-dimensional map to plot the actual meaning of the words. It is trying to understand that a 00:04:44.800 --> 00:04:49.120 message saying, you know, the financials are looking exceptionally tight this quarter, lives in the 00:04:49.120 --> 00:04:53.920 same mathematical neighborhood as the word budget, even though they share zero identical characters. 00:04:53.920 --> 00:04:58.960 That sounds incredibly resource intensive. It requires massive compute power and it relies 00:04:58.960 --> 00:05:03.920 on a highly complex, continuously updating pipeline. Every time a new email arrives, 00:05:03.920 --> 00:05:08.320 the system has to process it, generate those mathematical coordinates, and update the semantic 00:05:08.320 --> 00:05:14.560 map. Okay. So when the service change introduced a misconfiguration on July 22nd, that mathematical 00:05:14.560 --> 00:05:19.680 mapping pipeline just stopped processing new emails. Exactly. The map was never updated for 00:05:19.680 --> 00:05:25.600 anything arriving after that date. So the semantic index for those newer emails simply did not exist. 00:05:25.600 --> 00:05:31.680 It was a blank void. But as you deduced earlier, modern cloud systems are built with fallback 00:05:31.680 --> 00:05:37.200 mechanisms. When the semantic layer fails to find a contextual match for your prompt, the system 00:05:37.200 --> 00:05:43.200 recognizes the failure and awkwardly falls back to basic string matching, which perfectly explains 00:05:43.200 --> 00:05:49.360 the exact quote workaround. Yep. Like if I ask co-pilot, what did Sarah say about the Q3 numbers? 00:05:49.360 --> 00:05:54.400 It uses the semantic index. Search is the multi-dimensional map. Finds absolutely nothing after 00:05:54.400 --> 00:05:59.760 July 22nd and says, I don't know. But if I type the exact phrase Sarah wrote, Q3 numbers are down 00:05:59.760 --> 00:06:05.920 4%. The semantic search fails. The system panics defaults to C trial F mode searches the rocks, 00:06:05.920 --> 00:06:10.320 change database for that exact string of text. And boom, there is the email. It highlights how 00:06:10.320 --> 00:06:15.280 much architectural stress is created when you try to fuse probabilistic AI models onto 00:06:15.280 --> 00:06:20.880 deterministic legacy systems like exchange. You have this beautiful shiny AI layer sitting on 00:06:20.880 --> 00:06:25.359 top of decades old plumbing. And when the AI breaks, the illusion shatters instantly. 00:06:25.359 --> 00:06:32.159 Exactly. And it wasn't just back end indexing issues either. We also had incident CP 1447275. 00:06:32.159 --> 00:06:37.200 Oh, right. We're users trying to open the co-pilot analyst or researcher agents from the app store. 00:06:37.200 --> 00:06:43.359 We're getting redirected to entirely new blank chat windows instead of the actual functional 00:06:43.440 --> 00:06:49.040 agents, which is a classic state management and URL redirection bug. But it's wrapped up to look 00:06:49.040 --> 00:06:54.240 like an AI failure. Yeah. It goes to show that even when you are deploying cutting edge autonomous 00:06:54.240 --> 00:07:00.800 AI agents, a simple broken hyperlink in the app store can completely derail the user experience. 00:07:00.800 --> 00:07:05.120 It's the digital equivalent of walking into a multi-million dollar futuristic smart home, 00:07:05.120 --> 00:07:09.360 reaching for the front door and the door knob just falls off in your hand. A very apt comparison. 00:07:09.360 --> 00:07:12.720 The foundation has to be solid no matter how intelligent the software inside is. 00:07:12.720 --> 00:07:19.760 Right. So we just saw how AI failing to read user data creates a localized kind of chaos. 00:07:19.760 --> 00:07:24.480 But that localized chaos scales into an absolute enterprise nightmare when the system administrators 00:07:24.480 --> 00:07:29.520 themselves go blind. Oh, pipeline failures. Yeah. I mean, if co-pilot can't find an email, 00:07:29.520 --> 00:07:34.960 a user is annoyed. But if a telemetry pipeline stalls, an entire IT department is flying blind. 00:07:35.520 --> 00:07:39.440 Let's shift our focus from the user layer to the administrative data pipelines, 00:07:39.440 --> 00:07:43.599 because we saw some massive failures in reporting this week. We are really talking about the systemic 00:07:43.599 --> 00:07:49.120 problem of operational blindness here. First up, incident M one four six nine eight three six, 00:07:49.680 --> 00:07:55.280 admins logging into the Microsoft 365 admin center. We're looking at their Microsoft Teams usage 00:07:55.280 --> 00:08:00.960 reports and seeing zero active users. Just a terrifying thing to see zero chat messages. Yeah, 00:08:01.039 --> 00:08:05.039 zero channel messages, just a total flat line starting from August 25th. 00:08:05.039 --> 00:08:09.359 Then right next to that incident M one four seven one two four one one drive, 00:08:09.359 --> 00:08:14.240 SharePoint and group activity reports were all completely stale. The data just stopped updating 00:08:14.240 --> 00:08:19.359 Microsoft's incident report blamed upstream pipeline refresh errors. This points to a massive 00:08:19.359 --> 00:08:23.359 misunderstanding of how cloud interfaces actually work. I think a lot of people subconsciously 00:08:23.359 --> 00:08:27.840 assume that when an admin opens a cloud dashboard, the interface is running a real time query 00:08:27.840 --> 00:08:32.559 directly against the live production database. Like it's just doing a quick head count of the 00:08:32.559 --> 00:08:37.760 live rows in the database table at that exact second. Right. But doing that across millions of 00:08:37.760 --> 00:08:42.480 global tenants would absolutely crush the performance of a cloud service. Instead, 00:08:43.120 --> 00:08:49.760 these dashboards are fed by massive asynchronous data pipelines. Okay, asynchronous. Yeah. When you 00:08:49.760 --> 00:08:55.280 send a team's message, the system generates a telemetry event. That event gets queued, 00:08:55.360 --> 00:09:00.240 processed, aggregated with millions of others, and eventually pushed to the reporting databases. 00:09:00.240 --> 00:09:05.520 So it's basically a bucket brigade of data. It is a bucket brigade. Yes. But imagine the buckets 00:09:05.520 --> 00:09:10.480 are varying sizes and the people passing them are walking on a treadmill. That sounds precarious. 00:09:10.480 --> 00:09:15.680 It is. If one person slips, the buckets start piling up. The dashboard at the very end of the 00:09:15.680 --> 00:09:20.160 line doesn't know someone slipped. It just knows it hasn't received a bucket in five minutes. So to 00:09:20.160 --> 00:09:25.760 assume the water source, meaning the users has dried up, which brings us to the UX design choice. 00:09:25.760 --> 00:09:30.640 In the case of the OneDrive and SharePoint reports, the UI just froze and showed stale data. 00:09:30.640 --> 00:09:37.120 But in the case of the team's usage report, an error in the flow of telemetry data caused the UI 00:09:37.120 --> 00:09:43.920 to default to zero. I mean, defaulting a null value to a zero on an enterprise dashboard is 00:09:43.920 --> 00:09:48.959 guaranteed to induce a panic attack for whoever is on call. It is a known challenge in asynchronous 00:09:48.960 --> 00:09:54.080 systems. Developers hate null values because nulls break charts and graphs. So they tell the 00:09:54.080 --> 00:10:00.560 system, if you don't receive data, just plot a zero. But operationally, a zero is a terrifying 00:10:00.560 --> 00:10:06.160 lie. Because if my team's report says I have zero active users, I know immediately it's a bug. 00:10:06.160 --> 00:10:10.720 The business is still operating. People are still at their desks chatting. Exactly. But then we had 00:10:10.720 --> 00:10:17.920 incident mo 1457843. Entra ID audit logs suddenly stopped identifying who was deleting Viva engaged 00:10:17.920 --> 00:10:22.880 communities or Microsoft 365 groups. This one is huge. The log would just drop the name of the 00:10:22.880 --> 00:10:26.640 user who took the action. Yeah. If you're the one running the security operations center for 00:10:26.640 --> 00:10:31.040 your company, imagine getting a frantic call that a critical engineering group was deleted, 00:10:31.040 --> 00:10:35.920 opening your pristine, expensive audit dashboard, and just seeing a blank space where the culprit's 00:10:35.920 --> 00:10:41.520 name should be. It's chilling. Where do you even begin? That is a silent security and compliance 00:10:41.520 --> 00:10:46.880 nightmare. You've hit on the most concerning incident of the week. The root cause for that failure was 00:10:46.880 --> 00:10:53.040 a recent update designed to improve group deletions of Viva engaged communities. Wait, so they tried 00:10:53.040 --> 00:10:58.080 to make it faster or more efficient to delete a group? And in doing so, they broke the logging 00:10:58.080 --> 00:11:03.920 mechanism entirely. It is a classic architectural irony. In modern distributed systems, the system 00:11:03.920 --> 00:11:09.360 of action, the part that actually deletes the group is decoupled from the system of record, 00:11:09.360 --> 00:11:15.440 the audit log. Okay. By optimizing the deletion action, they inadvertently severed the asynchronous 00:11:15.440 --> 00:11:20.400 payload that carried the user's identity to the audit log. The deletion worked perfectly, 00:11:20.400 --> 00:11:25.600 but the receipt generated for it was blank. And the workaround was just brutal. Admins had to open a 00:11:25.600 --> 00:11:30.480 Microsoft support case with the deletion date and group details, just so support engineers could 00:11:30.480 --> 00:11:36.160 manually trace the initiating user through the raw service side backend logs, which takes days 00:11:36.160 --> 00:11:41.200 sometimes. Right. Imagine being in the middle of a high stakes insider threat investigation, 00:11:41.200 --> 00:11:45.680 and having to wait days for a vendor support ticket to tell you who just deleted a critical 00:11:45.680 --> 00:11:51.040 company asset. It completely undermines the concept of non-repudiation in your security posture. 00:11:51.040 --> 00:11:55.600 What do you mean? To briefly define that for context, non-repudiation means a user cannot deny 00:11:55.600 --> 00:12:01.040 having taken an action because the system has ironclad, mathematically proven proof that they did it. 00:12:01.040 --> 00:12:06.480 I see. If your logs drop the user's identity, they are no longer immutable or complete. 00:12:06.480 --> 00:12:12.160 Your compliance guarantees evaporate instantly. Man, that's rough. So we've seen how upstream 00:12:12.160 --> 00:12:17.760 failures turn admins blind. Let's look at the downstream impact of cloud state on local actions. 00:12:17.760 --> 00:12:22.800 The weird client bugs. Yeah. We are talking about how backend migrations and seemingly minor cloud 00:12:22.800 --> 00:12:28.560 updates are causing bizarre cascading regressions in basic decades old functionalities that run on 00:12:28.560 --> 00:12:35.600 your actual computer. Right. I'm looking at incident OP 1470668, Microsoft Excel. A code regression 00:12:35.600 --> 00:12:40.800 contained in a recent service update broke the ability to copy, paste, drag, or duplicate cells 00:12:40.800 --> 00:12:48.560 across Excel 2016, 2019, 2021, and 2024. A bug that sounds almost impossible on its face. 00:12:48.560 --> 00:12:52.240 I mean, I hear you on shared code bases and cloud connectivity, but frankly, 00:12:52.240 --> 00:12:57.200 that sounds like terrible architectural design. Why should a local clipboard operation in 00:12:57.200 --> 00:13:02.320 office in London require a permission slip from a cloud server in Virginia? It seems counterintuitive. 00:13:02.320 --> 00:13:07.360 Shouldn't core desktop functionalities like copying a cell be sandboxed? Like how does a modern 00:13:07.360 --> 00:13:13.760 cloud service update break a desktop client feature that has been working flawlessly since the 1990s? 00:13:13.760 --> 00:13:19.120 It is a phenomenal question. And it speaks to a massive misunderstanding about how modern desktop 00:13:19.120 --> 00:13:26.800 applications actually function. We like to think of Excel 2024 or even Excel 2016 as a standalone 00:13:26.800 --> 00:13:31.680 piece of software living isolated on your local hard drive. Right. Like the old CD-ROM days. 00:13:31.680 --> 00:13:36.959 Exactly. But in reality, modern desktop applications are deeply intimately entangled with cloud 00:13:36.959 --> 00:13:41.920 components through state management. Okay. And state management being the system's shared memory of 00:13:41.920 --> 00:13:47.199 what is true right now across all devices? Precisely. Think of state management as the 00:13:47.199 --> 00:13:51.680 system's memory of exactly what is happening in the application at any given microsecond. 00:13:52.239 --> 00:13:58.160 When you copy a cell in Excel today, you aren't just moving plain text from one local memory 00:13:58.160 --> 00:14:03.439 address to another. What else is moving? You are copying rich formatting, conditional logic, 00:14:03.439 --> 00:14:10.240 external data links, and crucially, real time co-authoring presence data. The local client is 00:14:10.240 --> 00:14:15.760 constantly syncing at state with backend services like Office Online Server to ensure that if someone 00:14:15.760 --> 00:14:20.480 else is viewing that document, they see your changes instantly. Oh, I see because you might be 00:14:20.480 --> 00:14:24.319 co-authoring a spreadsheet with three other people. Yeah. The clipboard isn't just local anymore. 00:14:24.400 --> 00:14:29.360 It's a networked event. Exactly. The code base is shared and synchronized to support that network 00:14:29.360 --> 00:14:34.960 event. If a cloud update changes how the backend expects to receive clipboard state data and the 00:14:34.960 --> 00:14:39.840 desktop client hasn't been updated to understand that new variable, the action fails. So they're 00:14:39.840 --> 00:14:44.720 out of sync. The local client simply doesn't know how to complete the operation because the server 00:14:44.720 --> 00:14:49.360 is speaking a slightly different language. It is a code regression born entirely from tightly 00:14:49.440 --> 00:14:54.640 coupled dependencies. And we saw this exact same shared state issue play out in two other major 00:14:54.640 --> 00:15:05.200 apps this week. In Microsoft Teams incident, TM 1459181, an update meant to quote, improve data pathways, 00:15:05.200 --> 00:15:10.160 ended up breaking the copy link button for shared channels. Oh, yeah, that one. The links being 00:15:10.160 --> 00:15:16.080 generated were completely malformed. Because a Teams channel link isn't just a static URL anymore. 00:15:16.080 --> 00:15:20.800 It contains tenant IDs, routing information, and permission tokens generated on the fly by 00:15:20.800 --> 00:15:26.240 that newly improved data pathway. So the improvement broke the token generation. Right. If the pathway 00:15:26.240 --> 00:15:31.520 has a bug, the state being handed down to the team's client is corrupted, resulting in a useless link. 00:15:31.520 --> 00:15:37.600 And then there was exchange online incident ex 1441488. Users were suddenly seeing unwanted 00:15:37.600 --> 00:15:41.840 add ins automatically appearing across all their outlet clients. That would be incredibly annoying. 00:15:41.920 --> 00:15:46.720 The add ins were supposed to be available to the organization, but explicitly disabled for the 00:15:46.720 --> 00:15:53.280 user. Why did they show up? Because a portion of the exchange sync pipeline blipped and returned 00:15:53.280 --> 00:15:59.040 incorrect enabled status values to the local outlet clients. This perfectly illustrates the 00:15:59.040 --> 00:16:04.240 danger of centralized state synchronization. The local outlet client essentially asks the cloud, 00:16:04.240 --> 00:16:09.680 Hey, what add ins should I show the user today? And the cloud just gets confused. The pipeline 00:16:09.760 --> 00:16:15.439 due to a synchronization bug replies, all of them. The local client doesn't have the context or 00:16:15.439 --> 00:16:20.880 the authority to know that's wrong. It just blindly obeys the state dictated by the cloud. 00:16:20.880 --> 00:16:27.199 It is wild to think about we built these incredibly powerful, globally distributed systems. But the 00:16:27.199 --> 00:16:32.400 trade off is that a single pipeline hiccup in a data center can force your local desktop app to 00:16:32.400 --> 00:16:36.800 install something it shouldn't or literally prevent you from dragging a cell in a spreadsheet. 00:16:36.800 --> 00:16:41.280 It blurs the line between the endpoint and the infrastructure. Your laptop is really just a 00:16:41.280 --> 00:16:47.040 viewport into a massive distributed state machine. Okay, it is definitely frustrating when copy paste 00:16:47.040 --> 00:16:52.240 breaks or you know, when an outlook add in clutter driven. But let's raise the space for our final 00:16:52.240 --> 00:16:58.000 segment. Let's do it. We've talked about state sync annoyances. What are the operational consequences 00:16:58.000 --> 00:17:04.000 when an infrastructure migration breaks security enforcement? Now we're moving from user frustration 00:17:04.000 --> 00:17:10.400 to genuine operational risk. Let's look at the Defender XDR conditional access app control failures. 00:17:10.400 --> 00:17:19.839 This was tracked across two incidents, DZ 1471 853 and DZ 1472 946. According to the source, 00:17:19.839 --> 00:17:24.319 a service infrastructure migration caused performance to drop below manageable thresholds. 00:17:24.319 --> 00:17:29.040 Basically, the infrastructure was overloaded to the point of choking. And the impact was severe. 00:17:29.040 --> 00:17:34.720 Users experienced high latency transient 5ix server errors. And crucially, this is the part 00:17:34.720 --> 00:17:39.680 that made my jaw drop. Some users accessed applications without the expected session monitoring and 00:17:39.680 --> 00:17:44.320 security controls in place. We should probably define what a transient 5 MgX server error is for 00:17:44.320 --> 00:17:49.280 the context of this failure. Please do. It essentially means the backend servers were basically throwing 00:17:49.280 --> 00:17:54.080 their hands up and saying, I am too overwhelmed to process this request right now. Please try again 00:17:54.080 --> 00:18:00.560 later. And when that happens to a security control, you face a critical architectural fork in the 00:18:00.560 --> 00:18:08.000 road. Wait, fail open versus fail closed. So if I visualizing this, fail closed is like a bank vault 00:18:08.000 --> 00:18:12.720 locking its massive doors when the power goes out. Good analogy. Nobody gets in. The business 00:18:12.720 --> 00:18:16.879 shuts down for the day, but the money is completely safe. Are you saying they chose to leave the 00:18:16.879 --> 00:18:22.000 vault doors wide open? You've summarized the dilemma of security architecture perfectly. 00:18:22.000 --> 00:18:26.560 Conditional access app control acts as a reverse proxy. It sits in the middle of the network path 00:18:26.560 --> 00:18:32.640 like a bouncer at a club inspecting traffic in real time. If you design that proxy to fail closed, 00:18:32.640 --> 00:18:37.520 then when the system is overwhelmed by a migration, all traffic is blocked. No one gets into their 00:18:37.520 --> 00:18:43.680 apps. It is incredibly secure, but you have just caused a massive company wide business outage. 00:18:43.680 --> 00:18:48.320 Which means the cure is worse than the disease for most businesses. They just can't afford a 00:18:48.399 --> 00:18:51.919 total work stoppage just because a security server is running slow. 00:18:51.919 --> 00:18:57.600 In many operational contexts, yes. So architects often design the system to fail open. 00:18:58.480 --> 00:19:04.240 When the proxy is overwhelmed and throwing those five X errors, it essentially waves the traffic 00:19:04.240 --> 00:19:09.120 through to keep the business running. Wow. You maintain availability, but you sacrifice 00:19:09.120 --> 00:19:14.480 confidentiality and integrity. The system intentionally bypasses the session monitoring 00:19:14.480 --> 00:19:18.640 controls to keep the traffic flowing, which is exactly what happened here. 00:19:18.640 --> 00:19:23.760 Users accessed apps without being monitored. But think about what that implies for the engineer on 00:19:23.760 --> 00:19:28.160 call. It's terrifying. If conditional access app control dropped its session monitoring during a 00:19:28.160 --> 00:19:33.280 migration because the servers are too busy, do we even know what users access during that window? 00:19:33.280 --> 00:19:38.560 That is a massive blind spot for a security operations center. It is a conscious architectural 00:19:38.639 --> 00:19:44.720 trade-off that Microsoft and frankly, any organization building network controls has to make. 00:19:44.720 --> 00:19:49.200 It's vital for you, thinking about your own network, to ask how your organization's 00:19:49.200 --> 00:19:53.280 architecture handles performance drops in the security layer. That's a great point. 00:19:53.280 --> 00:19:59.679 Do your firewalls fail open or closed under a DDoS attack? Do your identity providers grant 00:19:59.679 --> 00:20:05.040 access or deny it if the multi-factor authentication provider times out? It is the kind of thing you 00:20:05.040 --> 00:20:09.120 really want to know the answer to before an incident happens, not while you're reading a post 00:20:09.120 --> 00:20:14.240 incident reported at 2 a.m. Definitely. It's so easy to assume that a security tool is a solid wall, 00:20:14.800 --> 00:20:20.480 but in reality, it's a pressure valve. And sometimes under pressure, that valve has to open to prevent 00:20:20.480 --> 00:20:25.040 the entire pipeline from bursting. That is a brilliant analogy. And it ties all of these 00:20:25.040 --> 00:20:29.680 incidents together, really, whether we are talking about semantic search indices, building complex 00:20:29.680 --> 00:20:34.800 mathematical factors, asynchronous telemetry bucket brigades, shared cloud state for 00:20:34.800 --> 00:20:40.400 desktop apps, or conditional access reverse proxies, they are all invisible foundations. 00:20:40.400 --> 00:20:45.840 The plumbing behind the drywall of our digital offices. Yes. And as these invisible foundations 00:20:45.840 --> 00:20:51.520 become increasingly complex and deeply embedded into our daily workflows, a single point of failure 00:20:51.520 --> 00:20:56.080 no longer just takes an app offline. It's much deeper. It doesn't just give you a simple 404 00:20:56.080 --> 00:21:01.040 error page. It fundamentally alters our visibility and our understanding of what the system is 00:21:01.040 --> 00:21:06.480 actually doing. We are moving into an era where the system might still be technically up, but 00:21:06.480 --> 00:21:12.560 its behavior, its memory, and its security posture might be silently degrading. You can't just check 00:21:12.560 --> 00:21:17.840 if the server is pinging anymore. You have to ask, you know, is the state accurate? Is the pipeline 00:21:17.840 --> 00:21:23.360 flowing? Is the AI actually retaining its context? Exactly. And before we go, consider this broader 00:21:23.360 --> 00:21:30.080 implication. If our systems are now designed to silently bypass security or downgrade to older, 00:21:30.080 --> 00:21:35.040 less efficient logic just to keep the lights on during an overload, how long until autonomous AI 00:21:35.040 --> 00:21:40.879 systems learn to intentionally play dumb to save their own computing power? If falling back to 00:21:40.879 --> 00:21:46.159 keyword search saves massive server load, an AI optimizing for efficiency might choose to fail on 00:21:46.159 --> 00:21:50.719 purpose. It is a question of machine economics. We haven't even been done to answer. That is a 00:21:50.800 --> 00:21:56.320 deeply unsettling thought to leave on AI acting incompetent, just to save a few compute cycles. 00:21:56.960 --> 00:22:01.040 Well, that brings us to the end of our deep dive for the week to September 14 to 18, 00:22:01.040 --> 00:22:06.560 2026. We want to thank you, the listener, once again, for your time, your curiosity, and for 00:22:06.560 --> 00:22:10.880 coming along on this journey with us. You've been listening to Support Engineering Weekly from the 00:22:10.880 --> 00:22:15.360 Support Engineering Blog. For the full articles, references, and technical resources from today's 00:22:15.360 --> 00:22:21.040 discussion, visit the URL blog.sadhan.ch. Until next week, keep troubleshooting, 00:22:21.040 --> 00:22:26.639 keep learning, and keep engineering. Thank you for listening to support engineering weekly. 00:22:27.360 --> 00:22:32.639 This podcast is available on Spotify, Apple podcast, YouTube, and other major 00:22:32.639 --> 00:22:39.919 podcast platforms. Follow the show on your preferred platform and never miss an episode.