The issue with delayed check runs has been fully resolved and all check types are running normally from these locations. Our engineers have also updated the runner deploy process to help prevent issues like this in the future.
Posted Jun 17, 2019 - 18:33 EDT
Identified
We have identified the issue causing delayed runs in multiple locations. A code change was introduced to our deploy script that prevented runners from operating in one of our cloud providers. We have updated the code and are in the process of re-deploying runners to the faulty regions.
We have also brought up a previous runner version in these locations to cover the workload while the deploy runs. At this time, check runs in most affected regions are no longer delayed.
Posted Jun 17, 2019 - 17:07 EDT
Investigating
We are currently investigating delayed runs in several of our monitoring locations, beginning at approximately 3:30 pm ET (19:30 UTC). We are working to address the delay and will update here when the issue is resolved.
Posted Jun 17, 2019 - 16:05 EDT
This incident affected: Rigor Monitoring Network - Asia Pacific (New South Wales, Victoria, Osaka, Busan, Pune, India, Chennai, India), Rigor Monitoring Network - Europe (Cardiff), and Rigor Monitoring Network - North America (Toronto, Canada, Québec City, Canada, Illinois).