tag:status.watershedlrs.eu,2005:/historyWatershed LRS - EU Status - Incident History2024-03-28T07:51:24-05:00Watershed LRS - EUtag:status.watershedlrs.eu,2005:Incident/199676562024-02-21T00:00:56-06:002024-02-21T00:00:56-06:00Production (EU) RDS upgrade<p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>00:00</var> CST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>20:00</var> CST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>04:56</var> CST</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>20:56</var> CST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>20:00</var> CST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>15</var>, <var data-var='time'>13:49</var> CST</small><br><strong>Update</strong> - The scheduled maintenance window for Production (EU) has been postponed until Tuesday Feb 20 at 8pm CT (2024-02-21 02:00 UTC).</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>09:51</var> CST</small><br><strong>Scheduled</strong> - Production (EU) will be undergoing scheduled maintenance during this time.<br /><br />We have identified Web Servers, Analytics Database, and Reporting as possibly being affected. However, we do not expect any interruption to any of these components as the upgrade will be deployed on a rolling basis across our suite of application servers. We do not anticipate any issues with receiving inbound xAPI data during this maintenance window.</p>tag:status.watershedlrs.eu,2005:Incident/199136542024-02-08T14:06:30-06:002024-02-08T14:06:30-06:00Production (EU) RDS upgrade<p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>14:06</var> CST</small><br><strong>Completed</strong> - This maintenance window has been postponed. <br />A new maintenance window will be scheduled as soon as the time has been determined.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>10:15</var> CST</small><br><strong>Update</strong> - Production (EU) will be undergoing scheduled maintenance during this time.<br /><br />We have identified Web Servers, Analytics Database, and Reporting as possibly being affected. However, we do not expect any interruption to any of these components as the upgrade will be deployed on a rolling basis across our suite of application servers. Also, we do not anticipate any issues with receiving inbound xAPI data during this maintenance window.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>10:12</var> CST</small><br><strong>Scheduled</strong> - Production (EU) will be undergoing scheduled maintenance during this time.</p>tag:status.watershedlrs.eu,2005:Incident/180734712023-08-07T09:07:15-05:002023-08-07T09:07:15-05:00Reporting outage<p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>09:07</var> CDT</small><br><strong>Resolved</strong> - This issue has now been resolved.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>04:40</var> CDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Aug <var data-var='date'> 7</var>, <var data-var='time'>04:40</var> CDT</small><br><strong>Investigating</strong> - We are currently investigating an issue with reports not loading on our EU cluster.</p>tag:status.watershedlrs.eu,2005:Incident/150194092022-12-13T14:42:45-06:002022-12-13T14:42:45-06:00Watershed Reporting and data search not operational<p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>14:42</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>13:41</var> CST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>13:04</var> CST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.watershedlrs.eu,2005:Incident/147578812022-12-09T14:54:52-06:002022-12-09T14:54:52-06:00Reporting disruption<p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>14:54</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>10:49</var> CST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.watershedlrs.eu,2005:Incident/147537962022-12-09T10:14:12-06:002022-12-09T10:14:12-06:00Watershed Reporting and data search not operational<p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>10:14</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>09:30</var> CST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Dec <var data-var='date'> 9</var>, <var data-var='time'>09:30</var> CST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.watershedlrs.eu,2005:Incident/131557352022-11-15T15:49:46-06:002022-11-15T15:49:46-06:00Reporting disruption<p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>15:49</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>15</var>, <var data-var='time'>14:56</var> CST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.watershedlrs.eu,2005:Incident/110965472022-09-10T12:16:49-05:002022-09-10T12:16:49-05:00Database upgrade<p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>12:16</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>11:25</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>11:20</var> CDT</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.watershedlrs.eu,2005:Incident/105525362022-07-12T12:27:03-05:002022-07-12T12:27:03-05:00Dashboard permissions are not working<p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>12:27</var> CDT</small><br><strong>Resolved</strong> - This was assessed as a global issue but we've since discovered it is specific only to a single account, and is not a systemic issue.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>10:03</var> CDT</small><br><strong>Investigating</strong> - Currently dashboard permissions are not being applied to any Watershed user on our EU instance. This means that only users with Global Admin access will be able to view data in any reports.<br /><br />We apologise for this issue, and will update this as soon as we know any more.</p>tag:status.watershedlrs.eu,2005:Incident/102890092022-06-14T16:21:40-05:002022-06-14T16:21:40-05:00Reports not loading<p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>16:21</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>14</var>, <var data-var='time'>15:55</var> CDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:status.watershedlrs.eu,2005:Incident/72292472021-06-11T21:22:39-05:002021-06-11T21:22:39-05:00Reports not loading<p><small>Jun <var data-var='date'>11</var>, <var data-var='time'>21:22</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'>11</var>, <var data-var='time'>21:14</var> CDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jun <var data-var='date'>11</var>, <var data-var='time'>21:05</var> CDT</small><br><strong>Investigating</strong> - We are currently investigating an issue where reports are not loading.</p>tag:status.watershedlrs.eu,2005:Incident/57553112020-12-18T19:46:30-06:002020-12-18T19:46:30-06:00Scheduled maintenance - Dec 18, 2020<p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>19:46</var> CST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>19:16</var> CST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'> 8</var>, <var data-var='time'>14:06</var> CST</small><br><strong>Scheduled</strong> - We are upgrading systems on Friday December 18 between 1:16 and 1:46 AM UTC, with 1-2 minutes of actual downtime.</p>tag:status.watershedlrs.eu,2005:Incident/55867242020-11-17T07:30:00-06:002020-11-17T08:53:27-06:00Reports not loading<p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>07:30</var> CST</small><br><strong>Resolved</strong> - We are currently investigating an issue where reports are not loading.</p>tag:status.watershedlrs.eu,2005:Incident/41199202020-05-15T16:56:17-05:002020-05-15T16:56:17-05:00Watershed SSO intermittent outage<p><small>May <var data-var='date'>15</var>, <var data-var='time'>16:56</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>15</var>, <var data-var='time'>16:21</var> CDT</small><br><strong>Monitoring</strong> - A fix is out for this issue and we are monitoring it to ensure the partial SSO outage is over.</p><p><small>May <var data-var='date'>15</var>, <var data-var='time'>13:02</var> CDT</small><br><strong>Investigating</strong> - We are currently investigating an issue where some users cannot access their Watershed accounts via SSO. This does not affect all users, and no data/reporting will be lost.</p>tag:status.watershedlrs.eu,2005:Incident/39928952020-04-29T12:43:40-05:002020-04-29T12:43:40-05:00Custom Branding Not Working<p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>12:43</var> CDT</small><br><strong>Resolved</strong> - This issue is resolved</p><p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>10:31</var> CDT</small><br><strong>Identified</strong> - For some users, custom branding does not work in accounts. We've identified the cause and are working on a fix. No data or reporting is affected and this should be resolved quickly.</p>tag:status.watershedlrs.eu,2005:Incident/36744252020-03-21T13:35:30-05:002020-03-21T13:35:30-05:00Scheduled Maintenance<p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>13:35</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>13:25</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>11</var>, <var data-var='time'>13:26</var> CDT</small><br><strong>Scheduled</strong> - Watershed will be updating systems during this time, and Watershed will not be available. Also, incoming data will not be accepted. Existing imports will be paused and resume after the maintenance.</p>tag:status.watershedlrs.eu,2005:Incident/36115352020-02-25T12:09:17-06:002020-02-25T12:09:17-06:00Reports based on time not loading<p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>12:09</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved. Time based reports are now loading properly.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>10:52</var> CST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>10:33</var> CST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'>25</var>, <var data-var='time'>09:37</var> CST</small><br><strong>Investigating</strong> - We are currently investigating an issue with reports that use time based measures or dimensions not loading.<br /><br />There has been no data lost as due to this issue.</p>tag:status.watershedlrs.eu,2005:Incident/35899572020-02-20T17:10:28-06:002020-02-20T17:10:28-06:00Degraded Reporting on Watershed EU<p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>17:10</var> CST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>13:46</var> CST</small><br><strong>Monitoring</strong> - All statement data is now available for reporting. The earlier issues has been resolved and we are monitoring the system.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>11:43</var> CST</small><br><strong>Identified</strong> - The issues has been identified and we're currently working on implementing a solution.</p><p><small>Feb <var data-var='date'>20</var>, <var data-var='time'>10:26</var> CST</small><br><strong>Investigating</strong> - We're currently investigating an issue where new statements aren’t showing up in Watershed's EU environment. Existing data is safe and new data is stored, but it is not available to reporting.</p>tag:status.watershedlrs.eu,2005:Incident/30074042019-09-18T17:22:45-05:002019-09-18T17:22:45-05:00Unexpected Service Interruption<p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>17:22</var> CDT</small><br><strong>Resolved</strong> - This incident is now resolved. No data loss occurred during the service interruption.</p><p><small>Sep <var data-var='date'>18</var>, <var data-var='time'>09:13</var> CDT</small><br><strong>Monitoring</strong> - The system is now operational. A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'>17</var>, <var data-var='time'>16:18</var> CDT</small><br><strong>Update</strong> - The service has recovered, but we are still investigating the root cause. This issue will be updated with further details.</p><p><small>Sep <var data-var='date'>17</var>, <var data-var='time'>12:02</var> CDT</small><br><strong>Investigating</strong> - Engineers are investigating an unexpected service interruption</p>tag:status.watershedlrs.eu,2005:Incident/29922632019-09-14T00:00:38-05:002019-09-14T00:00:38-05:00System Upgrades<p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>00:00</var> CDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>23:00</var> CDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>13:37</var> CDT</small><br><strong>Scheduled</strong> - We will be performing system maintenance on Friday, September 13 from 11:00 PM CDT to 12:00 AM CDT. We expect no disruption in service.</p>tag:status.watershedlrs.eu,2005:Incident/24867612019-06-03T09:48:01-05:002019-06-03T09:48:01-05:00Custom Colors Not Applying To Accounts<p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>09:48</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>04:21</var> CDT</small><br><strong>Identified</strong> - Custom branding colors are currently not being applied, and users all will see Watershed blue. Custom chart colors and logos are unaffected. We are working to fix this as soon as possible.</p>tag:status.watershedlrs.eu,2005:Incident/24288262019-05-22T09:13:44-05:002019-05-22T09:13:44-05:00Reporting caches going into a read only mode from 04:30-13:30 UTC on 05/21/19<p><small>May <var data-var='date'>22</var>, <var data-var='time'>09:13</var> CDT</small><br><strong>Resolved</strong> - All data on EU is now available for reporting.</p><p><small>May <var data-var='date'>22</var>, <var data-var='time'>05:24</var> CDT</small><br><strong>Update</strong> - We have confirmed that full functionality should be restored by 11:00 UTC 05/22/19.</p><p><small>May <var data-var='date'>21</var>, <var data-var='time'>11:53</var> CDT</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue and aim for full functionality to be restored at around 13:30 UTC on 05/22/19.</p><p><small>May <var data-var='date'>21</var>, <var data-var='time'>05:14</var> CDT</small><br><strong>Identified</strong> - At 04:29 UTC on 05/21/19 the EU instance of Watershed began to not report on any data sent into Watershed sent after this time. The cause of this issue was resolved at 13:30 UTC, however the data is still not available in our reporting system. There has no data lost due to this issue.<br /> <br />This issue was caused by our reporting caches going into a read only mode due to an error with the storage attached to them.<br /><br />This outage only affected reporting on data sent after 04:30 UTC on the 05/21/19. Users were still able to login and run reports on all data logged before the issue during this period<br /><br />We have resolved the issue and are currently resynchronising the reporting caches to include the missing data, we estimate this process will complete by around 13:30 UTC on the 05/22/19.</p>