Microsoft’s Linux repositories were down for 18+ hours

Close-up photograph of a hand holding a toy penguin.
Enlarge / In 2017, Tux was unhappy that he had a Microsoft emblem on his chest. In 2021, he is largely unhappy that Microsoft’s repositories had been down for many of a day.

Jim Salter

Yesterday,—the repository from which Microsoft serves software program installers for Linux distributions together with CentOS, Debian, Fedora, OpenSUSE, and extra—went down arduous, and it stayed down for round 18 hours. The outage impacted customers making an attempt to put in .NET Core, Microsoft Groups, Microsoft SQL Server for Linux (sure, that is a factor) and extra—in addition to Azure‘s personal devops pipelines.

We first grew to become conscious of the issue Wednesday night once we noticed 404 errors within the output of apt replace on an Ubuntu workstation with Microsoft Groups put in. The outage is considerably better-documented at this .NET Core problem report on Github, with many customers from all around the globe sharing their experiences and theories.

The brief model is that all the repository cluster that serves all Linux packages for Microsoft was utterly down—issuing a spread of HTTP 404 (content material not discovered) and 500 (Inside Server Error) messages for any URL—for roughly 18 hours. Microsoft engineer Rahul Bhandari confirmed the outage roughly 5 hours after it was initially reported, with a cryptic remark in regards to the infrastructure crew “working into some house points.”

Eighteen hours after the problem was detailed, Bhandari mentioned that the mirrors had been as soon as once more out there—though with quickly degraded efficiency, possible attributable to chilly caches. On this replace, Bhandari mentioned that the unique explanation for the outage was “a regression in [apt repositories] throughout some characteristic migration work that resulted in these packages changing into unavailable on the mirrors.”

We’re nonetheless ready for a complete incident report, since Bhandari’s standing updates present clues however no actual explanations. The excellent news: We will affirm that is certainly up as soon as once more, and it’s serving packages because it ought to.