We have confirmed that full functionality should be restored by 11:00 UTC 05/22/19.
Posted May 22, 2019 - 05:24 CDT
Update
We are continuing to work on a fix for this issue and aim for full functionality to be restored at around 13:30 UTC on 05/22/19.
Posted May 21, 2019 - 11:53 CDT
Identified
At 04:29 UTC on 05/21/19 the EU instance of Watershed began to not report on any data sent into Watershed sent after this time. The cause of this issue was resolved at 13:30 UTC, however the data is still not available in our reporting system. There has no data lost due to this issue.
This issue was caused by our reporting caches going into a read only mode due to an error with the storage attached to them.
This outage only affected reporting on data sent after 04:30 UTC on the 05/21/19. Users were still able to login and run reports on all data logged before the issue during this period
We have resolved the issue and are currently resynchronising the reporting caches to include the missing data, we estimate this process will complete by around 13:30 UTC on the 05/22/19.