IRP Site Availability Issue
Summary of impact: From 12:07 GMT on 16 February 2021 until 13:30 GMT on 16 February 2021, several IRP site instances experienced intermittent connectivity and failure to respond symptoms. Sites may have been unable to transact sales.
Root cause: The cause was traced to a Microsoft datacentre in Europe. Microsoft Azure Engineers identified that a backend network device had become unhealthy and that traffic was not automatically rerouted, resulting in Azure Front Door & CDN requests to fail.
Mitigation: While Microsoft Azure engineers investigated, IRP mitigated the issue by removing clients from the Azure CDN where it controlled their DNS records. Microsoft Azure Engineers addressed the issue by manually removing the faulty device and rerouted traffic.
Resolution: Services began to be restored at 12:41 GMT on 16 February 2021 with IRP's mitigation steps, and were completely restored at 13:30 by Microsoft Azure's mitigation steps.
Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history/