WEBVTT 00:00:00.000 --> 00:00:05.140 You are listening to Support Engineering Weekly. 00:00:05.140 --> 00:00:09.240 You know, there is this illusion we all sort of operate under when we're building these 00:00:09.240 --> 00:00:10.680 modern cloud environments. 00:00:10.680 --> 00:00:11.680 Oh, absolutely. 00:00:11.680 --> 00:00:13.960 The whole neatly separated boxes thing. 00:00:13.960 --> 00:00:14.960 Right. 00:00:14.960 --> 00:00:18.800 We like to think of these platforms as like series of isolated containers. 00:00:18.800 --> 00:00:22.120 You have your email box, your file storage box, your chat box. 00:00:22.120 --> 00:00:26.080 And the theory is, if the printing box breaks, well, you know, at least you can still send 00:00:26.080 --> 00:00:27.080 an email. 00:00:27.080 --> 00:00:28.080 Exactly. 00:00:28.080 --> 00:00:32.839 You look at the reality of modern architectural dependencies and that illusion just completely 00:00:32.839 --> 00:00:33.839 shatters. 00:00:33.839 --> 00:00:34.839 It really does. 00:00:34.839 --> 00:00:40.120 You realize pretty quickly that a single core access configuration can take down everything. 00:00:40.120 --> 00:00:45.280 Yeah, everything from exchange to universal print to the UI of your AI assistant all 00:00:45.280 --> 00:00:47.160 at the exact same time. 00:00:47.160 --> 00:00:50.879 The blast radius is just not confined to a single app anymore. 00:00:50.879 --> 00:00:52.840 I'm Alex, by the way, and I'm Maya. 00:00:52.840 --> 00:00:55.040 Thank you so much for joining us on this deep dive. 00:00:55.040 --> 00:00:57.440 You are tuned in to support engineering weekly. 00:00:57.440 --> 00:01:02.720 And today our mission is to unpack a massive stack of Microsoft Message Center and service 00:01:02.720 --> 00:01:03.720 health updates. 00:01:03.720 --> 00:01:07.520 Yes, specifically from the week of September 7 through the 11th, 2026. 00:01:07.520 --> 00:01:08.520 Right. 00:01:08.520 --> 00:01:13.280 And we're looking at a week defined by cascading failures across totally disparate services, 00:01:13.280 --> 00:01:17.640 automation going just entirely off the rails and the heavy impact of internal network 00:01:17.640 --> 00:01:18.640 intercepts. 00:01:18.640 --> 00:01:19.640 It's a heavy week. 00:01:19.640 --> 00:01:24.280 We are going to break down the actual routing failures, the missing telemetry, and most 00:01:24.280 --> 00:01:28.840 importantly, the exact workarounds you need to keep your tenant afloat. 00:01:28.840 --> 00:01:30.000 So let's just jump right in. 00:01:30.000 --> 00:01:31.200 What's the big theme here? 00:01:31.200 --> 00:01:36.280 Well, the overarching theme of this particular stack of updates is what happens when the 00:01:36.280 --> 00:01:41.600 underlying invisible pipelines we take for granted suddenly glitch out. 00:01:41.600 --> 00:01:45.000 You're talking about things like authentication brokers and telemetry feeds, right? 00:01:45.000 --> 00:01:45.920 Exactly. 00:01:45.920 --> 00:01:51.120 When those backend components experience even a minor degradation, they manifest as these 00:01:51.120 --> 00:01:55.520 massive, highly confusing, front end outages for the end user. 00:01:55.520 --> 00:01:58.800 And they look like entirely unrelated problems, which is the worst part. 00:01:58.800 --> 00:02:03.240 So let's start by unpacking that massive cross service outage that span from August 31st 00:02:03.240 --> 00:02:04.240 to September 3rd. 00:02:04.240 --> 00:02:08.240 Yeah, this one perfectly illustrates that shattered illusion of separated boxes we were 00:02:08.240 --> 00:02:09.320 just talking about. 00:02:09.320 --> 00:02:10.320 For sure. 00:02:10.320 --> 00:02:20.240 We're looking at incident reports, mo 1465-074 and ex 1464-935. 00:02:20.240 --> 00:02:24.440 To set the stage for you, imagine you are a user logging in on a Tuesday morning. 00:02:24.440 --> 00:02:28.560 Just a normal Tuesday, you go to download an exchange email attachment and the download 00:02:28.560 --> 00:02:29.560 fails. 00:02:29.560 --> 00:02:30.560 Right. 00:02:30.560 --> 00:02:32.760 So you try to search across your Microsoft 365 tenant failure. 00:02:32.760 --> 00:02:36.400 You look at teams and everyone's present status is completely stale. 00:02:36.400 --> 00:02:42.280 And then you try to ask co-pilot to summarize an email, a standard grounding prompt, and 00:02:42.280 --> 00:02:43.440 that fails too. 00:02:43.440 --> 00:02:47.960 In desperation, you try to just print out a physical document and universal print jobs 00:02:47.960 --> 00:02:49.560 absolutely refuse to queue. 00:02:49.560 --> 00:02:54.240 I mean, listing those symptoms out all at once, it immediately sounds like a catastrophic 00:02:54.240 --> 00:02:55.920 systemic network failure. 00:02:55.920 --> 00:02:58.040 It sounds like the internet is broken. 00:02:58.040 --> 00:03:01.000 But the root cause was actually incredibly specific, wasn't it? 00:03:01.000 --> 00:03:02.520 Yeah, highly localized. 00:03:02.520 --> 00:03:06.160 It wasn't that the exchange databases were corrupted or that universal print servers 00:03:06.160 --> 00:03:07.840 were physically offline. 00:03:07.840 --> 00:03:12.440 The entire issue was traced back to a single core access configuration. 00:03:12.440 --> 00:03:17.720 That config acts as a shared broker across multiple Microsoft 365 services. 00:03:17.720 --> 00:03:21.480 Think of it like a corporate building's master keycard server going down. 00:03:21.480 --> 00:03:25.840 The physical doors to the email room, the SharePoint room, the printing room, they're 00:03:25.840 --> 00:03:27.160 all perfectly fine. 00:03:27.160 --> 00:03:29.760 The rooms themselves are fully operational. 00:03:29.760 --> 00:03:30.760 Right. 00:03:30.760 --> 00:03:35.680 But because the centralized badge reader system is offline, nobody can authenticate to get 00:03:35.680 --> 00:03:37.360 through any of those doors. 00:03:37.360 --> 00:03:42.120 Suddenly, you are locked out of every seemingly unrelated service at once. 00:03:42.120 --> 00:03:45.320 And to the user, it just looks like the whole building has burned down. 00:03:45.320 --> 00:03:46.320 Exactly. 00:03:46.320 --> 00:03:47.480 So why do we build it this way? 00:03:47.560 --> 00:03:52.640 Well, that is the architectural trade off of modern cloud identity and access management. 00:03:52.640 --> 00:03:58.120 Relying on centralized access brokers is fantastic for enforcing zero trust policies, conditional 00:03:58.120 --> 00:04:00.239 access and unified logging. 00:04:00.239 --> 00:04:04.000 Because you absolutely want one primary gatekeeper evaluating those signals. 00:04:04.000 --> 00:04:05.000 Right. 00:04:05.000 --> 00:04:10.600 But when that gatekeeper's configuration gets corrupted or it experiences a service degradation, 00:04:10.600 --> 00:04:15.000 the operational blast radius crosscuts entirely disparate services. 00:04:15.000 --> 00:04:19.920 Which is why we saw exchange web services failing, room fighters and outlook, just spinning 00:04:19.920 --> 00:04:21.920 infinitely while spinning forever. 00:04:21.920 --> 00:04:27.280 I mean, Microsoft purview policy hits failing to evaluate basic one drive file opening 00:04:27.280 --> 00:04:30.360 actions, timing out all simultaneously. 00:04:30.360 --> 00:04:35.880 So as a support engineer on the floor, how do you even begin triaging that? 00:04:35.880 --> 00:04:36.880 It's tough. 00:04:36.880 --> 00:04:42.880 You have tickets flooding in about offline printers, broken calendars, missing chat statuses. 00:04:42.880 --> 00:04:46.120 The noise to signal ratio is just astronomical. 00:04:46.120 --> 00:04:51.360 My first instinct would normally be to check local network connectivity or look for a massive 00:04:51.360 --> 00:04:53.080 ISP routing issue. 00:04:53.080 --> 00:04:56.080 But if you do that, you are burning hours chasing ghosts. 00:04:56.080 --> 00:04:57.080 Exactly. 00:04:57.080 --> 00:04:59.640 Pattern recognition is your only lifeline here. 00:04:59.640 --> 00:05:04.560 When completely unlinked services like teens presence and universal print fail simultaneously, 00:05:04.560 --> 00:05:08.280 you have to immediately stop troubleshooting the individual application layers. 00:05:08.280 --> 00:05:11.520 So you stop looking at the printer schooler, you stop looking at the team's local cash. 00:05:11.520 --> 00:05:12.520 Right. 00:05:12.520 --> 00:05:14.599 You have to look upstream to the common denominators. 00:05:14.599 --> 00:05:15.599 Yes. 00:05:15.599 --> 00:05:17.640 Authentication, authorization and core access brokers. 00:05:17.640 --> 00:05:21.820 So if your diagnostic telemetry shows that the requests are failing before they even 00:05:21.820 --> 00:05:23.880 reach the app layer's compute environment. 00:05:23.880 --> 00:05:26.000 Or that token issue and says timing out. 00:05:26.000 --> 00:05:27.000 Yeah. 00:05:27.000 --> 00:05:30.880 You know you are dealing with an access config issue. 00:05:30.880 --> 00:05:35.159 And you route that escalation directly to your identity management team rather than 00:05:35.159 --> 00:05:36.159 the app owners. 00:05:36.159 --> 00:05:37.159 I've bought on. 00:05:37.159 --> 00:05:41.159 That upstream dependency problem is a perfect bridge to the next major crisis from this 00:05:41.160 --> 00:05:42.600 week's updates. 00:05:42.600 --> 00:05:47.080 We just talked about authentication pipelines failing, which stops users at the door. 00:05:47.080 --> 00:05:51.400 But when a data pipeline goes down, the system actually starts making bad decisions on its 00:05:51.400 --> 00:05:52.400 own. 00:05:52.400 --> 00:05:53.400 Yes. 00:05:53.400 --> 00:05:57.280 Missing telemetry doesn't just mean you lose visibility on an admin dashboard. 00:05:57.280 --> 00:06:00.240 It actively triggers automated protective chaos. 00:06:00.240 --> 00:06:04.360 And we saw two distinct incidents that highlight this exact dynamic. 00:06:04.360 --> 00:06:11.160 The exchange external email delay, which is EX1467029. 00:06:11.160 --> 00:06:13.880 And the Microsoft 365 admin center usage report. 00:06:13.880 --> 00:06:16.280 Oh, the one showing zero active teams users, right? 00:06:16.280 --> 00:06:17.280 Yeah. 00:06:17.280 --> 00:06:18.280 Under MO1469836. 00:06:18.280 --> 00:06:19.280 That's the one. 00:06:19.280 --> 00:06:23.640 The mechanics behind both of these failures are deeply tied to how cloud platforms handle 00:06:23.640 --> 00:06:24.880 missing information. 00:06:24.880 --> 00:06:28.320 Let's focus heavily on the exchange issue for a second because this is where automated 00:06:28.320 --> 00:06:31.160 security systems actively caused the outage. 00:06:31.160 --> 00:06:32.160 Yeah. 00:06:32.160 --> 00:06:36.600 So users were trying to send and receive emails from external domains, just standard 00:06:36.600 --> 00:06:37.960 daily business flow. 00:06:37.960 --> 00:06:46.000 Instead, they were getting slammed with four 514.7.5 hundred server busy errors. 00:06:46.000 --> 00:06:49.720 Emails are being aggressively throttled and deferred by the system. 00:06:49.720 --> 00:06:53.680 So exchange online uses a multitude of data sources, right? 00:06:53.680 --> 00:06:58.360 It integrates heavily with Microsoft Defender for Office 365 to calculate dynamic sender 00:06:58.360 --> 00:06:59.360 reputation. 00:06:59.360 --> 00:07:04.400 It relies on an upstream distributed data platform to constantly feed it massive volumes 00:07:04.400 --> 00:07:07.280 of telemetry about IP behaviors across the globe. 00:07:07.280 --> 00:07:08.280 Exactly. 00:07:08.280 --> 00:07:11.440 But what happened here is that a recent internal change at Microsoft increased the size of 00:07:11.440 --> 00:07:13.960 read requests beyond intended physical thresholds. 00:07:13.960 --> 00:07:14.960 Oh, wow. 00:07:14.960 --> 00:07:19.600 So it overloaded that distributed data platform causing severe latency and drop connections. 00:07:19.600 --> 00:07:20.600 Right. 00:07:20.600 --> 00:07:24.680 The data platform is basically choking, meaning it can't serve up the full picture of center 00:07:24.680 --> 00:07:28.800 reputation to the services that evaluate inbound and outbound mail. 00:07:28.800 --> 00:07:31.720 But here is the critical failure point in the architecture. 00:07:31.720 --> 00:07:36.040 One of the sender based reputation processing jobs continue to run anyway. 00:07:36.040 --> 00:07:40.960 It executed on schedule, but it ran with substantially less input data than it normally 00:07:40.960 --> 00:07:41.960 expects. 00:07:41.960 --> 00:07:46.080 Because the upstream platform was choking, it only received a fraction of the historical 00:07:46.080 --> 00:07:47.560 IP data. 00:07:47.560 --> 00:07:50.520 But the job still reported a successful completion. 00:07:50.520 --> 00:07:51.720 That's the crazy part. 00:07:51.720 --> 00:07:56.400 There was no completeness check in the pipeline to flag that the data set was statistically 00:07:56.400 --> 00:07:57.700 insignificant. 00:07:57.700 --> 00:07:58.700 That is just wild. 00:07:58.700 --> 00:08:03.300 The system basically said, Hey, I finished processing the data without anyone asking, 00:08:03.300 --> 00:08:06.539 wait, did you process all of the data or just 5% of it? 00:08:06.539 --> 00:08:07.539 Right. 00:08:07.539 --> 00:08:12.219 And the downstream mail protection service then ingested this highly incomplete reputation 00:08:12.219 --> 00:08:16.700 data because it lacked the full historical context for these IP addresses. 00:08:16.700 --> 00:08:18.659 The algorithms just assumed the worst. 00:08:18.659 --> 00:08:19.659 Exactly. 00:08:19.659 --> 00:08:25.140 It assigned overly restrictive limits to legitimate, historically safe sending IPs. 00:08:25.140 --> 00:08:28.300 The volume based reputation control kicked in automatically. 00:08:28.300 --> 00:08:33.419 And suddenly normal business email was being throttled with those server busy errors. 00:08:33.419 --> 00:08:37.260 The protection mechanism functioned exactly as it was programmed to do based on the data 00:08:37.260 --> 00:08:38.380 it received. 00:08:38.380 --> 00:08:41.340 The problem was the data itself was truncated. 00:08:41.340 --> 00:08:44.699 I really have to push back on the design of this automated system though. 00:08:44.699 --> 00:08:45.699 How so? 00:08:45.699 --> 00:08:52.540 Why does an anti-span system default to aggressively throttling legitimate business flow rather 00:08:52.540 --> 00:08:56.380 than failing open when its own internal data pipeline breaks? 00:08:56.380 --> 00:08:57.900 Ah, I see what you mean. 00:08:57.900 --> 00:09:02.699 It feels like the system is essentially saying, I'm confused because I don't have my data, 00:09:02.699 --> 00:09:05.199 so I'm just going to shut down the business. 00:09:05.199 --> 00:09:11.540 It highlights a massive tension between strict security defaults and actual business continuity. 00:09:11.540 --> 00:09:14.500 It does, but it's the classic zero trust dilemma. 00:09:14.500 --> 00:09:18.699 In a modern security environment, systems are fundamentally designed to fail closed. 00:09:18.780 --> 00:09:24.460 If you can't definitively verify that an IP is safe based on current data, the architecture 00:09:24.460 --> 00:09:26.220 forces you to assume it's hostile. 00:09:26.220 --> 00:09:27.220 Right. 00:09:27.220 --> 00:09:31.740 If the system were to fail open because reputation data was missing, it could potentially allow 00:09:31.740 --> 00:09:38.180 a massive coordinated fishing or malware campaign to slip through the filters during that telemetry 00:09:38.180 --> 00:09:39.180 blind spot. 00:09:39.180 --> 00:09:44.500 So Microsoft by design, prioritizes strict protection over availability in that micromole. 00:09:44.500 --> 00:09:46.620 I understand the theory I do. 00:09:46.620 --> 00:09:50.140 And from an operational standpoint, when that completeness check is missing from the 00:09:50.140 --> 00:09:55.900 data pipeline, the system's protection literally becomes a self-inflicted denial of service 00:09:55.900 --> 00:09:56.900 attack. 00:09:56.900 --> 00:09:57.900 Yeah. 00:09:57.900 --> 00:10:01.980 You are effectively de-dossing your own tenants' mail flow because a backend reporting job 00:10:01.980 --> 00:10:04.180 didn't validate its data set size. 00:10:04.180 --> 00:10:06.620 It is absolutely a self-inflicted wound. 00:10:06.620 --> 00:10:12.300 And we saw a parallel upstream telemetry issue with those team's usage reports. 00:10:12.300 --> 00:10:18.699 Great admins were logging into the Microsoft 365 admin center and seeing zero active users, 00:10:18.699 --> 00:10:20.020 zero chat messages. 00:10:20.020 --> 00:10:23.020 And zero channel messages starting around late August. 00:10:23.020 --> 00:10:26.979 Seeing those zeros across the board probably gave a lot of compliance officers a heart 00:10:26.979 --> 00:10:27.979 attack. 00:10:27.979 --> 00:10:31.939 I can only imagine they probably assumed the entire logging infrastructure had been wiped. 00:10:31.939 --> 00:10:36.380 While that specific dashboard failure didn't block mail flow, it operated on the exact 00:10:36.380 --> 00:10:38.180 same underlying principle, didn't it? 00:10:38.180 --> 00:10:39.180 It did. 00:10:39.180 --> 00:10:43.579 An error occurred within the flow of service health telemetry data utilized by the reporting 00:10:43.579 --> 00:10:44.579 service. 00:10:44.579 --> 00:10:46.620 The upstream pipeline broke. 00:10:46.620 --> 00:10:51.339 So the downstream dashboard just reported zeros instead of caching the last known good 00:10:51.339 --> 00:10:53.500 state or displaying an error banner. 00:10:53.500 --> 00:10:57.819 It's a stark reminder to engineers that data pipelines are just as mission critical as 00:10:57.819 --> 00:10:59.540 your authentication pipelines. 00:10:59.540 --> 00:11:04.579 It really emphasizes that backend architectural failures have bizarre front-end symptoms. 00:11:04.580 --> 00:11:10.020 But let's shift gears from massive data pipeline failures to something a bit more insidious. 00:11:10.020 --> 00:11:12.020 Oh, you mean the Kade regressions? 00:11:12.020 --> 00:11:13.020 Yes. 00:11:13.020 --> 00:11:19.860 Sometimes a tiny, well-intentioned code update causes an incredibly specific localized operational 00:11:19.860 --> 00:11:21.020 headache. 00:11:21.020 --> 00:11:23.620 We tracked a trio of highly annoying regressions this week. 00:11:23.620 --> 00:11:28.100 Unwanted Outlook Add-ins, Malformed teams, shared channel links, and broken run-only 00:11:28.100 --> 00:11:29.660 teams workflows. 00:11:29.660 --> 00:11:33.540 These are the exact types of incidents that drive support teams absolutely crazy. 00:11:33.540 --> 00:11:38.020 At first glance, they look entirely like user error or corrupt local client caches. 00:11:38.020 --> 00:11:42.740 Let's examine EX1441448, the Outlook Add-in issue. 00:11:42.740 --> 00:11:46.939 Users suddenly saw administrative add-ins appearing and pinning themselves automatically across 00:11:46.939 --> 00:11:48.459 all their Outlook clients. 00:11:48.459 --> 00:11:53.500 And to be clear, these were add-ins that admins had deliberately configured an attendant to 00:11:53.500 --> 00:11:57.699 be merely available for users to install if they chose to. 00:11:57.699 --> 00:11:58.699 Right. 00:11:58.699 --> 00:12:00.900 They were never flagged to be auto-installed or mandated. 00:12:00.900 --> 00:12:01.900 So it broke. 00:12:01.900 --> 00:12:08.100 The mechanism of failure here is deeply tied to the Exchange Online Add-in Sync Pipeline. 00:12:08.100 --> 00:12:12.939 A specific portion of that pipeline stopped processing the add-in state information correctly. 00:12:12.939 --> 00:12:18.220 Oh, so when the Outlook client checked in to sync its configuration, the pipeline started 00:12:18.220 --> 00:12:21.980 returning incorrect, enabled status values in the payload. 00:12:21.980 --> 00:12:22.980 Exactly. 00:12:22.980 --> 00:12:27.579 So the client asks for the status of the add-in and the broken pipeline effectively lies, 00:12:27.579 --> 00:12:30.780 telling it, yes, this is fully enabled and mandated. 00:12:30.780 --> 00:12:33.780 I'm using the local client to auto-install it and clutter the ribbon. 00:12:33.780 --> 00:12:36.780 If I'm an engineer dealing with this right now, I don't want to wait for Microsoft to 00:12:36.780 --> 00:12:39.900 roll out a global patch to fix the sync state. 00:12:39.900 --> 00:12:40.900 Definitely not. 00:12:40.900 --> 00:12:44.220 And there is a specific operational workaround you can deploy immediately. 00:12:44.220 --> 00:12:48.980 You can use PowerShell to force a local tenant override bypassing that broken cloud sync 00:12:48.980 --> 00:12:49.980 state, right? 00:12:49.980 --> 00:12:51.500 Yes, you need to run. 00:12:51.500 --> 00:12:54.740 Set app-identity, then the application id. 00:12:54.740 --> 00:12:59.459 Dash organization app-private catalog dash default state for user disabled. 00:12:59.460 --> 00:13:05.340 Command explicitly forces the disabled state down to the clients, stripping out the unintentionally 00:13:05.340 --> 00:13:09.740 deployed add-ins until the permanent backend fix is fully saturated. 00:13:09.740 --> 00:13:13.940 Having that specific PowerShell parameter string is vital for containing the help desk 00:13:13.940 --> 00:13:14.940 ticket search. 00:13:14.940 --> 00:13:15.940 Absolutely. 00:13:15.940 --> 00:13:20.140 And moving over to the team's regressions, we saw two more interface level bugs. 00:13:20.140 --> 00:13:27.420 Under TM1459181, users couldn't join shared channels using certain generated links. 00:13:27.420 --> 00:13:31.339 But the bug was incredibly specific to how the user interacted with the UI. 00:13:31.339 --> 00:13:32.339 Yes. 00:13:32.339 --> 00:13:36.099 If they used the copy link option from the three dots menu at the top right of the channel, 00:13:36.099 --> 00:13:37.459 the link failed. 00:13:37.459 --> 00:13:41.860 But if they right-click the exact same channel in the left rail and copied the link there, 00:13:41.860 --> 00:13:42.860 it works perfectly. 00:13:42.860 --> 00:13:47.140 An update meant to optimize the data pathways for link generation actually caused the payloads 00:13:47.140 --> 00:13:51.060 generated from that specific top right UI element to be malformed. 00:13:51.060 --> 00:13:54.099 The routing parameters in the URL were essentially garbled. 00:13:54.100 --> 00:14:02.060 And similarly, in TM1440450, we saw a complex regression in how teams' processes shared 00:14:02.060 --> 00:14:03.780 workflow requests. 00:14:03.780 --> 00:14:07.580 Users trying to execute run-only workflows, where they have permission to trigger the 00:14:07.580 --> 00:14:12.940 automation, but lack permissions to own or edit the underlying code, were suddenly being 00:14:12.940 --> 00:14:15.660 blocked with a missing trigger URI error. 00:14:15.660 --> 00:14:20.300 If a user is getting a missing trigger URI error, my first instinct as a support engineer 00:14:20.300 --> 00:14:23.580 is to assume the flow was deleted or the endpoint changed. 00:14:23.660 --> 00:14:24.660 Right. 00:14:24.660 --> 00:14:29.180 But because this is a back-end regression in how teams processes the authorization token 00:14:29.180 --> 00:14:33.220 for run-only contexts, we have to do something unorthodox to route around it. 00:14:33.220 --> 00:14:37.820 The immediate practical action is to temporarily grant those affected users owner or co-owner 00:14:37.820 --> 00:14:40.300 permissions for that specific workflow. 00:14:40.300 --> 00:14:43.940 Elevating their privileges temporarily forces the workflow to execute under a different 00:14:43.940 --> 00:14:45.660 authorization context. 00:14:45.660 --> 00:14:50.020 Completely bypassing the broken run-only token validation logic that is currently failing. 00:14:50.100 --> 00:14:54.220 Obviously, you have to document that elevation and remember to revoke that access once the 00:14:54.220 --> 00:14:55.819 fix is fully deployed. 00:14:55.819 --> 00:14:58.699 But it will unblock your critical business processes today. 00:14:58.699 --> 00:15:03.020 It perfectly illustrates how modern support engineering often requires you to intentionally 00:15:03.020 --> 00:15:08.500 manipulate authorization contexts just to route around a code regression in a proprietary 00:15:08.500 --> 00:15:10.060 platform you don't control. 00:15:10.060 --> 00:15:12.140 It's basically digital duct tape. 00:15:12.140 --> 00:15:17.100 So let's transition from unintentional code regressions to situations where the disruption 00:15:17.100 --> 00:15:19.060 was actually an intentional action. 00:15:19.140 --> 00:15:20.140 Right. 00:15:20.140 --> 00:15:24.140 We had issues this week where the outage wasn't a bug in a deployment, but rather someone 00:15:24.140 --> 00:15:26.540 intentionally intercepting or altering the traffic. 00:15:26.540 --> 00:15:31.420 Either Microsoft testing their own systems or an organization's own network infrastructure 00:15:31.420 --> 00:15:33.140 stepping on the packets. 00:15:33.140 --> 00:15:41.180 This distinction is crucial for rapid triaging under CW 1469885. 00:15:41.180 --> 00:15:47.060 Users globally suddenly notice that the suggestion pills in Microsoft 365 co-pilot chat. 00:15:47.060 --> 00:15:51.300 Those little helpful contextual prompts that appear after a bot response to guide 00:15:51.300 --> 00:15:52.300 the conversation. 00:15:52.300 --> 00:15:53.739 Yeah, those just vanish. 00:15:53.739 --> 00:15:55.739 They were entirely stripped from the interface. 00:15:55.739 --> 00:15:59.500 Knowing how often these UIs are updated, you'd immediately assume a front-end deployment 00:15:59.500 --> 00:16:01.979 broke the CSS or the component rendering. 00:16:01.979 --> 00:16:03.619 Yeah, you would. 00:16:03.619 --> 00:16:07.859 But examining the service telemetry revealed that this was caused by an internal Microsoft 00:16:07.859 --> 00:16:08.859 drill. 00:16:08.859 --> 00:16:09.859 A drill. 00:16:09.859 --> 00:16:12.880 Yeah, the engineering teams were running a drill designed to test the resiliency of the 00:16:12.880 --> 00:16:14.500 underlying AI service. 00:16:14.500 --> 00:16:19.460 So that drill likely involved failing over massive back-end components. 00:16:19.460 --> 00:16:25.060 And during that failover, the service dropped non-critical UI payloads like the suggestion 00:16:25.060 --> 00:16:27.820 pills to maintain core chat functionality. 00:16:27.820 --> 00:16:28.820 Exactly. 00:16:28.820 --> 00:16:32.820 Once they halted the drill, the payload was restored and the impact was remediated. 00:16:32.820 --> 00:16:38.740 And then on the flip side of intentional disruption, we have MO1468437. 00:16:38.740 --> 00:16:41.220 Users couldn't edit virtual invites in Microsoft Teams. 00:16:41.220 --> 00:16:45.220 The interface just wouldn't load, spinning endlessly, or throwing connection errors. 00:16:45.220 --> 00:16:47.900 The alert Microsoft published for this was highly specific. 00:16:47.900 --> 00:16:53.140 It said user connections to Microsoft 365 domains are being TLS intercepted and decrypted 00:16:53.140 --> 00:16:54.940 by a network intermediary. 00:16:54.940 --> 00:17:00.220 It is absolutely maddening because TLS interception always disguises itself as a cloud outage to 00:17:00.220 --> 00:17:01.460 the end user. 00:17:01.460 --> 00:17:04.900 The user just sees a broken page in Blaine's Teams. 00:17:04.900 --> 00:17:09.579 As an engineer on the floor, how do you quickly prove that this is the corporate firewall breaking 00:17:09.579 --> 00:17:10.579 the connection? 00:17:10.579 --> 00:17:14.179 Rather than just another Microsoft regression, you don't want to waste hours on a support 00:17:14.179 --> 00:17:15.179 ticket. 00:17:15.179 --> 00:17:18.339 You have to look at the certificate chain and the network trace. 00:17:18.339 --> 00:17:22.619 If this were a Microsoft cloud outage like the co-pilot drill we just discussed, you would 00:17:22.619 --> 00:17:26.500 see a clean 200 OK HTTP response. 00:17:26.500 --> 00:17:30.460 But the specific JSON payload for the UI elements would be missing. 00:17:30.460 --> 00:17:35.860 However, with TLS interception, your corporate proxy is actively decrypting the traffic, 00:17:35.860 --> 00:17:39.860 encrypting it, and then re-encrypting it with its own internal certificate. 00:17:39.860 --> 00:17:44.379 And modern client applications like Teams often use strict certificate penning or highly 00:17:44.379 --> 00:17:46.219 specific WebSocket connections. 00:17:46.219 --> 00:17:50.020 Right, so when the Teams client detects that the certificate chain has been altered by 00:17:50.020 --> 00:17:53.860 the proxy, it forcefully drops the connection for security reasons. 00:17:53.860 --> 00:17:58.419 So if you pull a fiddler trace or check the browser's developer tools network tab, you 00:17:58.419 --> 00:18:00.179 won't see a missing payload. 00:18:00.179 --> 00:18:05.459 You will see SSL handshake failures or connection resets specifically on the Microsoft endpoints. 00:18:05.460 --> 00:18:09.380 And that proves the traffic isn't even securely reaching Microsoft's edge. 00:18:09.380 --> 00:18:13.780 It proves you can't assume every broken button is a cloud regression. 00:18:13.780 --> 00:18:17.620 Sometimes it's your own security appliance trying to be a little too helpful and breaking 00:18:17.620 --> 00:18:18.900 the encryption chain. 00:18:18.900 --> 00:18:22.340 It requires a completely different troubleshooting methodology. 00:18:22.340 --> 00:18:26.980 You aren't checking service health bashboards, you are checking proxy bypass rules. 00:18:26.980 --> 00:18:27.980 That's a great point. 00:18:27.980 --> 00:18:32.060 Now that we've contained this week's active fires, we need to talk about the ones brewing 00:18:32.060 --> 00:18:33.060 for next month. 00:18:33.060 --> 00:18:34.780 Oh, the AI updates. 00:18:34.780 --> 00:18:40.220 Specifically, how upcoming AI feature updates are going to violently collide with your current 00:18:40.220 --> 00:18:42.060 data governance frameworks. 00:18:42.060 --> 00:18:45.660 We have two major message center updates to prepare for. 00:18:45.660 --> 00:18:52.220 MC11222978, detailing the new co-pilot dashboard day level exports. 00:18:52.220 --> 00:18:57.540 And MC1146821, introducing the edge for business rewrite feature. 00:18:57.540 --> 00:19:02.020 The co-pilot dashboard update is a massive leap forward for organizations desperately 00:19:02.020 --> 00:19:06.300 trying to measure the actual return on investment for generative AI. 00:19:06.300 --> 00:19:09.940 Because previously analytics were limited in granularity, right? 00:19:09.940 --> 00:19:13.100 Making it really hard to see daily adoption trends. 00:19:13.100 --> 00:19:14.100 Exactly. 00:19:14.100 --> 00:19:20.220 Now, users with company level access can export 28 days of de-identified user data. 00:19:20.220 --> 00:19:25.139 This role level data is aggregated daily, allowing architects to tactically track adoption 00:19:25.139 --> 00:19:29.620 patterns and honestly optimize their heavy licensing costs. 00:19:29.620 --> 00:19:33.020 But there are strict guardrails on pulling that level of data. 00:19:33.020 --> 00:19:34.020 Right. 00:19:34.020 --> 00:19:40.139 First, the tenant requires a minimum of 50 Microsoft 365 co-pilot licenses to even activate the 00:19:40.139 --> 00:19:41.139 feature. 00:19:41.139 --> 00:19:46.300 Second, and critical for your governance teams, access to this new day level export utilizes 00:19:46.300 --> 00:19:52.060 the exact same Viva feature access management or VFAM controls as the existing user week 00:19:52.060 --> 00:19:53.060 export. 00:19:53.060 --> 00:19:57.540 So if your compliance team has already restricted access to analytics using VFAM policies, those 00:19:57.620 --> 00:20:01.100 restrictions will automatically inherit and apply to this new day level data. 00:20:01.100 --> 00:20:02.100 Right. 00:20:02.100 --> 00:20:04.260 You don't have to build a parallel governance structure. 00:20:04.260 --> 00:20:06.740 That automatic inheritance is such a relief. 00:20:06.740 --> 00:20:10.260 But the update that really requires immediate attention is the rewrite feature coming to 00:20:10.260 --> 00:20:11.260 edge for business. 00:20:11.260 --> 00:20:15.659 Powered by co-pilot chat, it lets users highlight text directly in the browser and ask the AI 00:20:15.659 --> 00:20:18.659 to rewrite it, adjust the tone or summarize it. 00:20:18.659 --> 00:20:21.860 It's rolling out between late September and late October. 00:20:21.860 --> 00:20:26.620 But there is a massive governance implication buried in the technical documentation. 00:20:26.620 --> 00:20:32.300 It revolves entirely around how AI interacts with data loss prevention or DLP integration. 00:20:32.300 --> 00:20:38.179 The documentation explicitly states that if Microsoft 365 co-pilot is restricted from 00:20:38.179 --> 00:20:43.840 accessing page content due to existing DLP policies, the rewrite feature will simply 00:20:43.840 --> 00:20:45.820 not function on that content. 00:20:45.820 --> 00:20:49.459 Think of it like putting a genius assistant in a sensory deprivation tank. 00:20:49.459 --> 00:20:50.860 I love that analogy. 00:20:50.860 --> 00:20:55.760 If your DLP rules are configured to block browser scraping or data extraction on a specific 00:20:55.760 --> 00:21:00.860 internal site, an internet wiki or a sensitive web app, you are essentially blocking the 00:21:00.860 --> 00:21:02.720 AI's eyes and ears. 00:21:02.720 --> 00:21:05.760 It cannot read the DOM so it cannot help you rewrite the text. 00:21:05.760 --> 00:21:08.480 This is a fundamental paradigm shift for security architects. 00:21:08.480 --> 00:21:13.120 We used to think of DLP purely as a mechanism to block file uploads or prevent copying text 00:21:13.120 --> 00:21:14.120 to a clipboard. 00:21:14.120 --> 00:21:17.480 Now DLP dictates the functional boundaries of generative AI. 00:21:17.480 --> 00:21:20.440 AI will only work as well as your data policies allow. 00:21:20.440 --> 00:21:26.600 If you have overly aggressive legacy DLP policies that blindly block all browser scraping 00:21:26.600 --> 00:21:31.880 on all internal domains, your users are going to flood the help desk complaining that the 00:21:31.880 --> 00:21:34.760 new edge rewrite feature is broken. 00:21:34.760 --> 00:21:36.560 But it won't be broken. 00:21:36.560 --> 00:21:41.040 The application will be functioning exactly as governed by your security team, which means 00:21:41.040 --> 00:21:45.240 you need to act right now before this hits the production ring in late September. 00:21:45.240 --> 00:21:49.840 You need to audit your DLP configurations to understand exactly how they intersect with 00:21:49.840 --> 00:21:51.919 browser content access. 00:21:51.919 --> 00:21:56.320 You also need to verify your inline compose enabled policy setting, which manages the 00:21:56.320 --> 00:21:59.080 edge rewrite feature at the administrative level. 00:21:59.080 --> 00:22:02.000 You have to map out which internal domains are restricted. 00:22:02.000 --> 00:22:06.080 So you aren't caught off guard when users submit tickets claiming the AI refuses to 00:22:06.080 --> 00:22:10.040 rewrite a paragraph on their highly classified internal project site. 00:22:10.040 --> 00:22:13.760 Modern DLP rules can no longer be broad brush network blocks. 00:22:13.760 --> 00:22:18.639 They must be highly context aware, carefully balancing stringent data security with the 00:22:18.640 --> 00:22:22.160 required functional inputs of generative AI tools. 00:22:22.160 --> 00:22:25.760 Bringing all of these threads together, there are three critical technical lessons you should 00:22:25.760 --> 00:22:28.080 take away from this week's deep dive. 00:22:28.080 --> 00:22:33.640 Number one, unified access architectures require you to prepare for wide blast radius cross 00:22:33.640 --> 00:22:35.400 service outages. 00:22:35.400 --> 00:22:39.520 When the master keycard server experiences degradation, everything from exchange web 00:22:39.520 --> 00:22:43.840 services to universal print cues will fail simultaneously. 00:22:43.840 --> 00:22:47.280 You have to triage the identity layer, not the application layer. 00:22:47.360 --> 00:22:52.480 Number two, automated security systems are fundamentally designed to fail closed when 00:22:52.480 --> 00:22:54.920 their internal telemetry pipelines break. 00:22:54.920 --> 00:22:59.960 As we saw with the Exchange 451 server busy throttling, a broken upstream data pipeline 00:22:59.960 --> 00:23:05.040 can lead to massive false positive throttling downstream, effectively allowing a backend 00:23:05.040 --> 00:23:09.200 analytics glitch to block legitimate critical business traffic. 00:23:09.200 --> 00:23:13.680 And number three, copilot's new capabilities are heavily reliant on your organization's 00:23:13.680 --> 00:23:15.520 existing governance frameworks. 00:23:15.520 --> 00:23:20.520 Either as VFAM policies controlling who can see the new 28 day analytics exports or legacy 00:23:20.520 --> 00:23:24.760 DLP policies dictating what the edge rewrite tool is physically allowed to read in the 00:23:24.760 --> 00:23:25.760 browser. 00:23:25.760 --> 00:23:30.040 Your AI deployment is entirely dependent on the precision of your data policies. 00:23:30.040 --> 00:23:34.440 Every single failure and feature we discussed today traces back to complex dependencies, 00:23:34.440 --> 00:23:40.160 whether they are architectural routing, data driven telemetry, or policy based access controls, 00:23:40.160 --> 00:23:43.680 which leaves us with a final thought to ponder as you look at your own tenant. 00:23:43.680 --> 00:23:48.160 For routine internal backend resiliency drill can accidentally draw off the payload 00:23:48.160 --> 00:23:53.840 for front end UI components, like making copilot suggestion pills vanish into thin air. 00:23:53.840 --> 00:23:58.880 How resilient are these new AI interfaces to underlying network fluctuations compared 00:23:58.880 --> 00:24:01.360 to traditional static UI tools? 00:24:01.360 --> 00:24:05.360 We are entering an era where the interface itself is generated dynamically based on real 00:24:05.360 --> 00:24:09.920 time prompts, which means the UI you click on is just as fragile as the data pipeline 00:24:09.920 --> 00:24:11.600 running behind it. 00:24:11.600 --> 00:24:15.360 You have been listening to Support engineering weekly from the Support Engineering Blog. 00:24:15.360 --> 00:24:19.719 For the full articles, references, and technical resources from today's discussion, visit the 00:24:19.719 --> 00:24:22.959 URL blog.sadhan.ch. 00:24:22.959 --> 00:24:27.919 Until next week, keep troubleshooting, keep learning, and keep engineering. 00:24:27.919 --> 00:24:31.360 Thank you for listening to Support Engineering Weekly. 00:24:31.360 --> 00:24:36.360 This podcast is available on Spotify, Apple podcast, YouTube, and other major 00:24:36.360 --> 00:24:37.639 podcast platforms. 00:24:37.639 --> 00:24:42.040 Follow the show on your preferred platform and never miss an episode.