IRP PAAS Status

Intermittent connectivity

Summary of impact: Between 07:50 GMT and 12:18 GMT on 14 June 2022, monitoring systems reported intermittent IRP website availability. IRP website performance and/or availability may have been impacted.

Root cause: Azure Front Door, on which IRP websites are dependent, experienced a surge in traffic resulting in connectivity issues. Errors being reported by IRP's monitoring included certificate issues and connection resets/timeouts.

Party Responsible: Azure - IRP websites are dependent on Front Door. Certificates and network connectivity are managed by Azure as part of Front Door.

Mitigation: Azure re-balanced traffic and scaled-out the service to meet the extra unexpected demand.

Resolution: Azure re-balanced traffic, scaled-out the Front Door service and continues to monitor for further impact. Azure is continuing to investigate the full root cause to prevent future occurrences.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history/

IRP Websites Unavailable

Summary of impact: At 19:07 GMT on 8 June 2022, monitoring systems reported all IRP websites were inaccessible. Services were restored at 19:55 GMT.

Root cause: This was been traced to an issue within Azure. Azure Support were made aware and investigated. Specifically this was an issue with DNS records controlled by Azure on which IRP services were dependent.

Mitigation: None

Resolution: Issue was resolved by Azure

Further Information:

Intermittent Availability

Summary of impact: Between 10:00 GMT and 11:00 GMT on 30 May 2022, monitoring systems started to report timeouts when connecting to IRP sites for users located in Europe.

Root cause: Azure engineers identified potential issues with 3rd party ISP's

Mitigation: None required. IRP services were operating normally during this period.

Resolution: None required. IRP services were operating normally during this period.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history/

IRP Site Availability Issue

Summary of impact: From 22:22 GMT on 1 April 2021 until 23:40 GMT on 1 April 2021, several IRP site instances experienced intermittent connectivity. Sites may have been unable to transact sales.

Root cause: The cause was traced to disruption of Azure DNS services, which had a knock-on effect to other downstream Azure services such as Azure CDN.  The cause of the disruption was identified as an unexpected spike in DNS traffic.

Mitigation: Azure engineers investigated and deployed mechanisms to absorb the spike in DNS traffic.

Resolution: Services were completely restored at 23:40 GMT by Microsoft Azure's mitigation steps.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history/

IRP Site Availability Issue

Summary of impact: From 12:07 GMT on 16 February 2021 until 13:30 GMT on 16 February 2021, several IRP site instances experienced intermittent connectivity and failure to respond symptoms. Sites may have been unable to transact sales.

Root cause: The cause was traced to a Microsoft datacentre in Europe. Microsoft Azure Engineers identified that a backend network device had become unhealthy and that traffic was not automatically rerouted, resulting in Azure Front Door & CDN requests to fail.

Mitigation: While Microsoft Azure engineers investigated, IRP mitigated the issue by removing clients from the Azure CDN where it controlled their DNS records. Microsoft Azure Engineers addressed the issue by manually removing the faulty device and rerouted traffic.

Resolution: Services began to be restored at 12:41 GMT on 16 February 2021 with IRP's mitigation steps, and were completely restored at 13:30 by Microsoft Azure's mitigation steps.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history/

IRP Site Availability Issue

Summary of impact: From 18:41 GMT on 19 June 2018 until 00:55 GMT on 20 June 2018, several IRP site instances experienced failure to respond symptoms. Sites could not transact any sales.

Root cause: The cause was traced to a Microsoft datacentre in North Europe i.e. Dublin. Microsoft Azure Engineers identified an underlying temperature issue in the datacentre triggered an infrastructure alert, which in turn caused structured precautionary shutdown of a subset of Storage and Network devices to ensure hardware and data integrity

Mitigation: Microsoft Azure Engineers addressed the temperature issue, and performed a structured recovery of the affected devices and any downstream services that were affected

Resolution: Services began to be restored at 00:27 GMT on 20 June 2018, and were completely restored by 00:55 GMT on 20 June 2018