Microsoft 365 Cloud Service Outage Disrupts Users Worldwide

Cloud Security, Critical Infrastructure Security, Endpoint Security

‘Network Change’ Tied to Service Disruption Now Rolled Back, Tech Giant Reports

Matthew J. Schwartz (euroinfosec) •
January 25, 2023

Image: Pixabay

Microsoft has blamed an internal network configuration change for a series of outages that have globally disrupted access to its Azure cloud services, including web-based Outlook and Microsoft Teams.

See Also: Live Webinar | Navigating the Difficulties of Patching OT

“Any user serviced by the affected infrastructure may be unable to access multiple Microsoft 365 services,” Microsoft warned Wednesday morning in a Microsoft 365 service degradation alert.

The outage impacted a wide range of services, including not just Microsoft Teams and Exchange Online, but also SharePoint Online and OneDrive for Business. Other affected services included Microsoft 365 admin portal, Microsoft Intune endpoint management, as well as Microsoft Defender for Cloud Apps, Identity and Endpoint. Microsoft said the problem didn’t just impact direct access to its services, but also how information was flowing between its data centers.

Microsoft, around 08:00 am UTC, reported that it had identified the likely problem, which it described as “a wide-area networking routing change.” That suggests the change broke domain name system integrity for its services.

“We are seeing dropped packets and high latency to resources in Azure at the moment,” one UK-based systems engineer posted to Mastodon shortly thereafter.

About one hour later, the technology reported: “We’ve rolled back a network change that we believe is causing impact. We’re monitoring the service as the rollback takes effect.”

Following the change, Microsoft said more users had regained access to their Microsoft 365 services.

“We’re also connecting the service to additional infrastructure to expedite the recovery process,” Microsoft tweeted.

The Downdetector website, which crowdsources reports of service outages, said reports of disruptions began to emerge around 07:00 am UTC. Also impacted were consumer-focused services such as Minecraft and Xbox Live.

Microsoft’s cloud-based services are used by many of the world’s largest companies, meaning the outage likely impacted millions of individuals.

Microsoft’s Azure status page displayed outages across Africa, the Americas, Asia-Pacific, Europe and the Middle East. The only unaffected region was China, including Azure Government for China.

Many users took to Twitter to report outages, in some cases via the #Outlookdown hashtag.

“I guess Teams and Outlook going down means I can finally have a break from people contacting me,” one user tweeted. “But seriously, Teams and Outlook not working is really disrupting my day.”

Microsoft last experienced a major outage in June 2022, when access to Microsoft 365 was disrupted. So too was access to its Azure cloud computing environment, which is second only to Amazon Web Services in terms of the size of its user base.

Other large providers have not been immune either. Also in June 2022, Google reported that a serious outage that disrupted access for users in the Middle East, and led to an increase in latency for users in Europe and Asia, traced to a physical infrastructure pboelm.

“The outage was triggered by two simultaneous fiber cuts within our Middle East network. This affected the end-to-end path for several submarine cables, reducing capacity for many telecom and technology companies, including Google,” it reported.

In June 2021, Facebook reported being “unreachable” for many users for nearly 3 hours, saying it was “the worst outage we’ve had in over four years.”

Facebook’s engineering team said the problem that caused the service involved disruption a flaw in its automated system for verifying database integrity. Even when fixed, the flaw had created a “feedback loop” that continued to disrupt its databases. To fix the problem, Facebook had to turn its automated database integrity verification service off and back on again.

.

Leave a Comment