Delayed Check Runs in multiple Azure-based locations
Incident Report for Rigor
Resolved
This incident has been resolved, and our cloud provider has identified and mitigated the root cause.
Posted Sep 23, 2020 - 08:46 EDT
Monitoring
Multiple fixes have been implemented to mitigate the on-going issue with one of our cloud providers. While they continue to investigate and mitigate on their end, we are going to continue to monitor closely, however normal operations of locations have resumed.
Posted Aug 21, 2020 - 09:00 EDT
Update
We are continuing to investigate this issue.
Posted Jul 06, 2020 - 13:38 EDT
Update
We have implemented some stability improvements to improve test execution in our Illinois location. We are continuing to work with our provider to identify the root cause of the issue and will continue to provide updates here. Check timings are not affected by this issue, but runs in this location may be delayed.
Posted Jun 30, 2020 - 10:30 EDT
Update
We are continuing to take corrective actions, and are working with our cloud providers on a resolution.
Posted Jun 26, 2020 - 16:13 EDT
Update
We are experience a long running issue in our Illinois location in which there may be periodic windows of delayed check execution.

Timings and data collected during runs in this location are still accurate and not impacted by this issue.

We are working on both short term and long term fixes to address this issue, as well as working with our downstream providers to resolve the underlying problem.

This issue will remain open until we can implement a fix to stabilize the location.
Posted Jun 24, 2020 - 11:01 EDT
Update
We are continuing to investigate this issue.
Posted Jun 23, 2020 - 11:04 EDT
Investigating
We are experiencing delayed test execution in our Illinois location. We are working to identify the cause of the issue.

We will provide updates as soon as possible.
Posted Jun 23, 2020 - 09:35 EDT
This incident affected: Rigor Monitoring Network - Europe (Netherlands) and Rigor Monitoring Network - North America (Toronto, Canada, Illinois, Texas).