IRP Commerce Cloud Status

IRP Services Intermittent

Summary of impact: Between approx 09:00 and 12:04, access to IRP services may be intermittent, with brief periods of poor performance or connectivity issues.

Root cause: A transient networking issue caused intermittent connectivity issues for a set of Microsoft resources when accessing from the the UK region.

Party Responsible: Azure

Mitigation: A VPN routing traffic outside of the UK to access IRP services may allow successful connections.

Resolution: Azure engineers have resolved this issue.

Further Information: Further information will be added as it becomes available. Azure status history can be found at https://status.azure.com/en-us/status/history.

IRP Site Availability Issue

Summary of impact: Between approx 11:38 - 15:05 and 15:47 - 20:39, access to IRP services was intermittent, with brief periods of poor performance or connectivity issues.

Root cause: A DDOS event caused Azure Front Door services performance to degrade leading to latency and timout spikes, and increased intermittent errors. IRP Commerce services are built upon the Azure Front Door service and so were impacted.

Party Responsible: Azure

Mitigation: Azure engineers took mitigation steps around 15:10. There was improvement in services for a period before they began degrading again, so further mitigations were deployed around 19:00.

Resolution: Azure engineers have now migitgated this incident and declared it resolved.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history.

Azure hosting issue

Summary of impact: Between approx 08:30 and 13:55, a subset of IRP services became unresponsive and inaccessible. The majority of IRP services were unaffected and continued to operate normally

Root cause: Azure identified and unexpected increase in pressure on the infrastructure dataplane Virtual Machines, resulting in timeout errors and high latency.

Party Responsible: Azure

Mitigation: Azure engineers responded by manually scaling out the affected instances and implementing the required configuration changes, which has restored services.

Resolution: Azure have mitigated the issues and are continuing their investigations to establish the full root cause and determine how to prevent future occurrences.

Further Information: Further information will be added as it becomes available.

IRP App Issue

Summary of impact: The IRP Insights App is unavailable and is throwing an exception

Root cause: The Leap year is not being handled in the IRP software

Party Responsible: IRP

Mitigation: Holding page has been put up 

Resolution: A code fix needs to be released - but cannot be rushed as the build is part way. Resolved 10:00am February 2024

Further Information: Further information will be added as it becomes available.  The fix requires a release and this will be done in the next few days. 

IRP Site Availability Issue

Summary of impact: Between 07:45 and 18:15, access to IRP websites may have encountered intermittent timeouts and/or poor performance.

Root cause: Azure conducted an investigation, which identified a subset of Point-of-Presence (POP's) that were throwing errors and unable to process requests.

Party Responsible: Azure

Mitigation: Uncaching the IRP and retries were successful in many cases.

Resolution: Azure identified the POP's affected and moved to healthy alternatives, returning services to a healthy state.

Further Information: Further information will be added as it becomes available.

IRP Site Availability Issue

Summary of impact: Between 07:10 GMT and 08:55 GMT, access to IRP websites was disrupted, with intermittent timeouts and/or poor performance.

Root cause: Azure acknowledged there was an issue and began an investigation. The investigation revealed a network connectivity issue occurring with devices across the Microsoft Wide Area Network (WAN). This impacted connectivity between clients on the Internet to Azure, as well as connectivity between services in datacenters, and ExpressRoute connections.

Party Responsible: Azure

Mitigation: For users who were experiencing connectivity issues, retries were likely to be successful.

Resolution: Azure identified a recent change to the Microsoft WAN as the underlying cause, and have taken steps to roll back this change.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history

IRP Site Availability Issue

Summary of impact: Between 17:15 GMT and 21:19 GMT on 7 Sept 2022, IRP websites experienced intermittent access.

Root cause: Azure Front Door, on which IRP websites are dependent, experienced connectivity issues. Errors reported by IRP's monitoring included certificate issues and connection resets/timeouts.  Azure engineers are investigating a spike in traffic as a potential cause, which caused multiple environments managing and handling traffic to go offline.  Auto-recovery mitigations were unable to recover fully and required manual intervention.

Party Responsible: Azure

Mitigation: For users who were experiencing connectivity issues, retries were likely to be successful.

Resolution: Azure engineers manually recovered a number of Front Door nodes/environments to ensure all were returned to a healthy state.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history

Card Payment Entry Hampering Checkout Progress

Summary of impact: After 23:20 on 30 August, customers on some IRP's have been unable to place orders using card payments.  Submitting details to pay by card was resulting in the page reloading, and no further progress being possible. Payment methods such as Paypal, Amazon Pay, and Apple Pay were unaffected.

Root cause: An update by Google to their Tag Manager product has caused widespread problems on all major eCommerce platforms.

Party Responsible: Google

Mitigation: IRP has disabled Google Tag Manager across IRP Merchants.

Resolution: Google have yet to update on what was changed and why the changes have caused such widespread damage.

Further Information: More information will be provided as it becomes available.

Intermittent connectivity

Summary of impact: Between 07:50 GMT and 12:18 GMT on 14 June 2022, monitoring systems reported intermittent IRP website availability. IRP website performance and/or availability may have been impacted.

Root cause: Azure Front Door, on which IRP websites are dependent, experienced a surge in traffic resulting in connectivity issues. Errors being reported by IRP's monitoring included certificate issues and connection resets/timeouts.

Party Responsible: Azure - IRP websites are dependent on Front Door. Certificates and network connectivity are managed by Azure as part of Front Door.

Mitigation: Azure re-balanced traffic and scaled-out the service to meet the extra unexpected demand.

Resolution: Azure re-balanced traffic, scaled-out the Front Door service and continues to monitor for further impact. Azure is continuing to investigate the full root cause to prevent future occurrences.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history/

IRP Websites Unavailable

Summary of impact: At 19:07 GMT on 8 June 2022, monitoring systems reported all IRP websites were inaccessible. Services were restored at 19:55 GMT.

Root cause: This was been traced to an issue within Azure. Azure Support were made aware and investigated. Specifically this was an issue with DNS records controlled by Azure on which IRP services were dependent.

Party Responsible: Azure

Mitigation: None

Resolution: Issue was resolved by Azure

Further Information:

Intermittent Availability

Summary of impact: Between 10:00 GMT and 11:00 GMT on 30 May 2022, monitoring systems started to report timeouts when connecting to IRP sites for users located in Europe.

Root cause: Azure engineers identified potential issues with 3rd party ISP's

Party Responsible: Azure

Mitigation: None required. IRP services were operating normally during this period.

Resolution: None required. IRP services were operating normally during this period.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history/

IRP Site Availability Issue

Summary of impact: From 22:22 GMT on 1 April 2021 until 23:40 GMT on 1 April 2021, several IRP site instances experienced intermittent connectivity. Sites may have been unable to transact sales.

Root cause: The cause was traced to disruption of Azure DNS services, which had a knock-on effect to other downstream Azure services such as Azure CDN.  The cause of the disruption was identified as an unexpected spike in DNS traffic.

Party Responsible: Azure

Mitigation: Azure engineers investigated and deployed mechanisms to absorb the spike in DNS traffic.

Resolution: Services were completely restored at 23:40 GMT by Microsoft Azure's mitigation steps.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history/

IRP Site Availability Issue

Summary of impact: From 12:07 GMT on 16 February 2021 until 13:30 GMT on 16 February 2021, several IRP site instances experienced intermittent connectivity and failure to respond symptoms. Sites may have been unable to transact sales.

Root cause: The cause was traced to a Microsoft datacentre in Europe. Microsoft Azure Engineers identified that a backend network device had become unhealthy and that traffic was not automatically rerouted, resulting in Azure Front Door & CDN requests to fail.

Party Responsible: Azure

Mitigation: While Microsoft Azure engineers investigated, IRP mitigated the issue by removing Merchants from the Azure CDN where it controlled their DNS records. Microsoft Azure Engineers addressed the issue by manually removing the faulty device and rerouted traffic.

Resolution: Services began to be restored at 12:41 GMT on 16 February 2021 with IRP's mitigation steps, and were completely restored at 13:30 by Microsoft Azure's mitigation steps.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history/

IRP Site Availability Issue

Summary of impact: From 18:41 GMT on 19 June 2018 until 00:55 GMT on 20 June 2018, several IRP site instances experienced failure to respond symptoms. Sites could not transact any sales.

Root cause: The cause was traced to a Microsoft datacentre in North Europe i.e. Dublin. Microsoft Azure Engineers identified an underlying temperature issue in the datacentre triggered an infrastructure alert, which in turn caused structured precautionary shutdown of a subset of Storage and Network devices to ensure hardware and data integrity

Party Responsible: Azure

Mitigation: Microsoft Azure Engineers addressed the temperature issue, and performed a structured recovery of the affected devices and any downstream services that were affected

Resolution: Services began to be restored at 00:27 GMT on 20 June 2018, and were completely restored by 00:55 GMT on 20 June 2018