IRP COMMERCE CLOUD STATUS
99.97% Uptime last 3 Months

Summary of impact: Starting around 15:43 UTC, IRP monitoring began alerting for disrupted access to IRP merchant sites.  This started with increased latencies, progressing to timeouts and errors.  This lasted until approx 19:07 UTC, after which services began to recover, however some access was intermittent while the recovery process was ongoing.  IRP Merchant services were fully recovered by 22:40 UTC.

Root cause: Azure confirmed degradation of the Azure Front Door (CDN) service, which IRP uses to host client services.  This outage also affected other Azure and Microsoft services that have dependencies on Azure Front Door including the Azure Portal, which IRP use for management and monitoring of client services.  This was a global outage, and Azure have confirmed that a tenant configuration triggered this outage after their protection mechanisms failed to detect the at fault deployment.

Party Responsible: Azure

Mitigation: IRP actively monitored the situation and were in discussions with our hosting partners.  Azure actively took actions to restore services including blocking further configuration changes and performing a roll back to a last known good state.  Azure confirmed that the rollback completed successfully and that they were recovering nodes and re-routing traffic through healthy nodes using a "gradual by design" process to ensure stability and prevent overloading of recovered services.  During this process of recovery some traffic may have ended up at unhealthy nodes, so IRP merchant services continued to be intermittent until the process was complete.  Full recovery was expected to be completed around 23:20 UTC, but IRP monitoring recorded no further disruption beyond 22:40 UTC.  Azure have also implemented additional validation and rollback controls to prevent a reoccurrence.

Resolution: Azure actively took steps to restore services.

Further Information: More information will be provided as it becomes available.  The Azure status history can be found at https://status.azure.com/en-us/status/history.

Summary of impact: Starting around 08:55 BST, IRP monitoring began indicating that access to IRP services may be intermittent. Users could have experienced failed connections, slow connections and/or browser warnings due to certificate errors.  This lasted until approx 17:00 BST, at which time Azure declared the issue mitigated.

Root cause: Azure Front Door (CDN), which IRP uses to host client services, experienced a signficant degradation in service (approx 30% capacity) predominently affecting Europe and Africa. The root cause was identified as a latent platform bug which triggered crashes in a subset of environments.  This placed extra load on the remaining resources causing reduced availability and increased latency across Europe, Middle East and Africa

Party Responsible: Azure

Mitigation: No action by merchants were required. IRP were actively monitoring the situation.  Azure acknowledged the degradation of service and actively began taking actions to restore services once the root cause was identified.  Users could have experienced periods of higher than normal latency as recovery efforts were ongoing.

Resolution: Azure worked to restore services after identifying the root cause.

Further Information: More information will be provided as it becomes available.  Azure status history can be found at https://status.azure.com/en-us/status/history.

Summary of impact: Starting around 09:42 BST, one IRP service was disrupted until 10:50 BST

Root cause: A power failure during maintenance in the Azure datacentre hosting IRP services affected several Azure services, including those used by IRP Commerce

Party Responsible: Azure

Mitigation: IRP services have been restored.

Resolution: IRP services have been restored.

Further Information: Azure have mitigated the incident and are performing a thorough review with the aim of preventing future occurrences

Summary of impact: Starting around 15:55, IRP services were disrupted for between 4 and 9 minutes for most IRP clients

Root cause: IRP services lost connectivity with their database for at least 4 minutes

Party Responsible: Azure

Mitigation: IRP services have been restored.

Resolution: IRP services have been restored.

Further Information: Azure have identified an issue with their SQL service and have prepared an update to prevent future occurrences

Summary of impact: IRP Services could become disrupted due to infrastructure damage caused by Storm Éowyn

Root cause: Storm Éowyn has triggered red weather warnings across Ireland and United Kingdom. This means that there could be flying debris that poses a risk to life, and possible infrastructure damage. IRP services are hosted in Azure data centres, one of which is located in Ireland and therefore may be affected by the storm.

Party Responsible: Azure

Mitigation: Azure data centres will have redundant systems in place for power and connectivity, but disruption could still be possible. IRP personnel will be on standby and working to restore services as quickly as possible if there is disruption.

Resolution: More information will be provided as it becomes available

Summary of impact: Starting 14:30 on 23rd Jan 2025, access to IRP services may be intermittent, with periods of poor performance or connectivity issues.

Root cause: IRP Commerce are performing maintenance and releasing IRP updates which may result in those services restarting and causing periods of disruption.

Party Responsible: IRP Commerce

Resolution: Periods of disruption are expected to brief.

Summary of impact: Between 11:00 and 12:00, access to IRP services may be intermittent, with periods of poor performance or connectivity issues.

Root cause: IRP Commerce are making a change to database services configuration, which may result in those services restarting and causing periods of disruption.

Party Responsible: IRP Commerce

Resolution: Periods of disruption are expected to brief.

Summary of impact: Between approx 09:00 and 12:04, access to IRP services may be intermittent, with brief periods of poor performance or connectivity issues.

Root cause: A transient networking issue caused intermittent connectivity issues for a set of Microsoft resources when accessing from the the UK region.

Party Responsible: Azure

Mitigation: A VPN routing traffic outside of the UK to access IRP services may allow successful connections.

Resolution: Azure engineers have resolved this issue.

Further Information: Further information will be added as it becomes available. Azure status history can be found at https://status.azure.com/en-us/status/history.

Summary of impact: Between approx 11:38 - 15:05 and 15:47 - 20:39, access to IRP services was intermittent, with brief periods of poor performance or connectivity issues.

Root cause: A DDOS event caused Azure Front Door services performance to degrade leading to latency and timout spikes, and increased intermittent errors. IRP Commerce services are built upon the Azure Front Door service and so were impacted.

Party Responsible: Azure

Mitigation: Azure engineers took mitigation steps around 15:10. There was improvement in services for a period before they began degrading again, so further mitigations were deployed around 19:00.

Resolution: Azure engineers have now migitgated this incident and declared it resolved.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history.

Summary of impact: Between approx 08:30 and 13:55, a subset of IRP services became unresponsive and inaccessible. The majority of IRP services were unaffected and continued to operate normally.

Root cause: Azure identified and unexpected increase in pressure on the infrastructure dataplane Virtual Machines, resulting in timeout errors and high latency.

Party Responsible: Azure

Mitigation: Azure engineers responded by manually scaling out the affected instances and implementing the required configuration changes, which has restored services.

Resolution: Azure have mitigated the issues and are continuing their investigations to establish the full root cause and determine how to prevent future occurrences.

Further Information: Further information will be added as it becomes available.

Summary of impact: The IRP Insights App is unavailable and is throwing an exception.

Root cause: The Leap year is not being handled in the IRP software.

Party Responsible: IRP

Mitigation: Holding page has been put up.

Resolution: A code fix needs to be released - but cannot be rushed as the build is part way. Resolved 10:00am February 2024.

Further Information: Further information will be added as it becomes available. The fix requires a release and this will be done in the next few days.

Summary of impact: Between 07:45 and 18:15, access to IRP websites may have encountered intermittent timeouts and/or poor performance.

Root cause: Azure conducted an investigation, which identified a subset of Point-of-Presence (POP's) that were throwing errors and unable to process requests.

Party Responsible: Azure

Mitigation: Uncaching the IRP and retries were successful in many cases.

Resolution: Azure identified the POP's affected and moved to healthy alternatives, returning services to a healthy state.

Further Information: Further information will be added as it becomes available.

Summary of impact: Between 07:10 GMT and 08:55 GMT, access to IRP websites was disrupted, with intermittent timeouts and/or poor performance.

Root cause: Azure acknowledged there was an issue and began an investigation. The investigation revealed a network connectivity issue occurring with devices across the Microsoft Wide Area Network (WAN). This impacted connectivity between clients on the Internet to Azure, as well as connectivity between services in datacenters, and ExpressRoute connections.

Party Responsible: Azure

Mitigation: For users who were experiencing connectivity issues, retries were likely to be successful.

Resolution: Azure identified a recent change to the Microsoft WAN as the underlying cause, and have taken steps to roll back this change.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history.

Summary of impact: Between 17:15 GMT and 21:19 GMT on 7 Sept 2022, IRP websites experienced intermittent access.

Root cause: Azure Front Door, on which IRP websites are dependent, experienced connectivity issues. Errors reported by IRP's monitoring included certificate issues and connection resets/timeouts. Azure engineers are investigating a spike in traffic as a potential cause, which caused multiple environments managing and handling traffic to go offline. Auto-recovery mitigations were unable to recover fully and required manual intervention.

Party Responsible: Azure

Mitigation: For users who were experiencing connectivity issues, retries were likely to be successful.

Resolution: Azure engineers manually recovered a number of Front Door nodes/environments to ensure all were returned to a healthy state.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history.

Summary of impact: After 23:20 on 30 August, customers on some IRP's have been unable to place orders using card payments. Submitting details to pay by card was resulting in the page reloading, and no further progress being possible. Payment methods such as Paypal, Amazon Pay, and Apple Pay were unaffected.

Root cause: An update by Google to their Tag Manager product has caused widespread problems on all major eCommerce platforms.

Party Responsible: Google

Mitigation: IRP has disabled Google Tag Manager across IRP Merchants.

Resolution: Google have yet to update on what was changed and why the changes have caused such widespread damage.

Further Information: More information will be provided as it becomes available.

Summary of impact: Between 07:50 GMT and 12:18 GMT on 14 June 2022, monitoring systems reported intermittent IRP website availability. IRP website performance and/or availability may have been impacted.

Root cause: Azure Front Door, on which IRP websites are dependent, experienced a surge in traffic resulting in connectivity issues. Errors being reported by IRP's monitoring included certificate issues and connection resets/timeouts.

Party Responsible: Azure

Mitigation: Azure re-balanced traffic and scaled-out the service to meet the extra unexpected demand.

Resolution: Azure re-balanced traffic, scaled-out the Front Door service and continues to monitor for further impact. Azure is continuing to investigate the full root cause to prevent future occurrences.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history/

Summary of impact: At 19:07 GMT on 8 June 2022, monitoring systems reported all IRP websites were inaccessible. Services were restored at 19:55 GMT.

Root cause: This was been traced to an issue within Azure. Azure Support were made aware and investigated. Specifically this was an issue with DNS records controlled by Azure on which IRP services were dependent.

Party Responsible: Azure

Mitigation: None.

Resolution: Issue was resolved by Azure.

Summary of impact: Between 10:00 GMT and 11:00 GMT on 30 May 2022, monitoring systems started to report timeouts when connecting to IRP sites for users located in Europe.

Root cause: Azure engineers identified potential issues with 3rd party ISP's

Party Responsible: Azure

Mitigation: None required. IRP services were operating normally during this period.

Resolution: None required. IRP services were operating normally during this period.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history/

Summary of impact: From 22:22 GMT on 1 April 2021 until 23:40 GMT on 1 April 2021, several IRP site instances experienced intermittent connectivity. Sites may have been unable to transact sales.

Root cause: The cause was traced to disruption of Azure DNS services, which had a knock-on effect to other downstream Azure services such as Azure CDN. The cause of the disruption was identified as an unexpected spike in DNS traffic.

Party Responsible: Azure

Mitigation: Azure engineers investigated and deployed mechanisms to absorb the spike in DNS traffic.

Resolution: Services were completely restored at 23:40 GMT by Microsoft Azure's mitigation steps.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history/

Summary of impact: From 12:07 GMT on 16 February 2021 until 13:30 GMT on 16 February 2021, several IRP site instances experienced intermittent connectivity and failure to respond symptoms. Sites may have been unable to transact sales.

Root cause: The cause was traced to a Microsoft datacentre in Europe. Microsoft Azure Engineers identified that a backend network device had become unhealthy and that traffic was not automatically rerouted, resulting in Azure Front Door & CDN requests to fail.

Party Responsible: Azure

Mitigation: While Microsoft Azure engineers investigated, IRP mitigated the issue by removing Merchants from the Azure CDN where it controlled their DNS records. Microsoft Azure Engineers addressed the issue by manually removing the faulty device and rerouted traffic.

Resolution: Services began to be restored at 12:41 GMT on 16 February 2021 with IRP's mitigation steps, and were completely restored at 13:30 by Microsoft Azure's mitigation steps.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history/

Summary of impact: From 18:41 GMT on 19 June 2018 until 00:55 GMT on 20 June 2018, several IRP site instances experienced failure to respond symptoms. Sites could not transact any sales.

Root cause: The cause was traced to a Microsoft datacentre in North Europe i.e. Dublin. Microsoft Azure Engineers identified an underlying temperature issue in the datacentre triggered an infrastructure alert, which in turn caused structured precautionary shutdown of a subset of Storage and Network devices to ensure hardware and data integrity.

Party Responsible: Azure

Mitigation: Microsoft Azure Engineers addressed the temperature issue, and performed a structured recovery of the affected devices and any downstream services that were affected.

Resolution: Services began to be restored at 00:27 GMT on 20 June 2018, and were completely restored by 00:55 GMT on 20 June 2018.

Further Information: Azure status history can be found at https://status.azure.com/en-us/status/history/