IRP PAAS Status

IRP Site Availability Issue

Summary of impact: From 22:22 GMT on 1 April 2021 until 23:40 GMT on 1 April 2021, several IRP site instances experienced intermittent connectivity. Sites may have been unable to transact sales.

Root cause: The cause was traced to disruption of Azure DNS services, which had a knock-on effect to other downstream Azure services such as Azure CDN.  The cause of the disruption was identified as an unexpected spike in DNS traffic.

Mitigation: Azure engineers investigated and deployed mechanisms to absorb the spike in DNS traffic.

Resolution: Services were completely restored at 23:40 GMT by Microsoft Azure's mitigation steps.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history/

IRP Site Availability Issue

Summary of impact: From 12:07 GMT on 16 February 2021 until 13:30 GMT on 16 February 2021, several IRP site instances experienced intermittent connectivity and failure to respond symptoms. Sites may have been unable to transact sales.

Root cause: The cause was traced to a Microsoft datacentre in Europe. Microsoft Azure Engineers identified that a backend network device had become unhealthy and that traffic was not automatically rerouted, resulting in Azure Front Door & CDN requests to fail.

Mitigation: While Microsoft Azure engineers investigated, IRP mitigated the issue by removing clients from the Azure CDN where it controlled their DNS records. Microsoft Azure Engineers addressed the issue by manually removing the faulty device and rerouted traffic.

Resolution: Services began to be restored at 12:41 GMT on 16 February 2021 with IRP's mitigation steps, and were completely restored at 13:30 by Microsoft Azure's mitigation steps.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history/

IRP Site Availability Issue

Summary of impact: From 18:41 GMT on 19 June 2018 until 00:55 GMT on 20 June 2018, several IRP site instances experienced failure to respond symptoms. Sites could not transact any sales.

Root cause: The cause was traced to a Microsoft datacentre in North Europe i.e. Dublin. Microsoft Azure Engineers identified an underlying temperature issue in the datacentre triggered an infrastructure alert, which in turn caused structured precautionary shutdown of a subset of Storage and Network devices to ensure hardware and data integrity

Mitigation: Microsoft Azure Engineers addressed the temperature issue, and performed a structured recovery of the affected devices and any downstream services that were affected

Resolution: Services began to be restored at 00:27 GMT on 20 June 2018, and were completely restored by 00:55 GMT on 20 June 2018