tag:status.innovalog.com,2005:/historyJMWE Cloud Status - Incident History2024-03-07T09:24:11-05:00JMWE Cloudtag:status.innovalog.com,2005:Incident/194737502023-12-16T09:00:00-05:002023-12-18T15:31:28-05:00JMWE Auto-Scaler Crash<p><small>Dec <var data-var='date'>16</var>, <var data-var='time'>09:00</var> EST</small><br><strong>Resolved</strong> - We experienced a disruption in message processing due to a JMWE auto-scaler crash on the backend. All servers scaled down to 0, causing a backlog. The issue is now resolved, and processing has resumed.</p>tag:status.innovalog.com,2005:Incident/126039622022-10-24T16:07:30-04:002022-10-24T16:07:30-04:00JMWE Cloud Server is down. The team is working on resolving the issue.<p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>16:07</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>15:31</var> EDT</small><br><strong>Monitoring</strong> - The issue is resolved. We will continue to monitor the situation for some more time before closing this incident. Our apologies for the inconvinence.</p><p><small>Oct <var data-var='date'>24</var>, <var data-var='time'>15:18</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.innovalog.com,2005:Incident/94005712022-02-24T12:28:11-05:002022-02-24T12:28:11-05:00JMWE outage caused by a widespread Heroku outage<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>12:28</var> EST</small><br><strong>Resolved</strong> - The incident is now resolved.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>12:19</var> EST</small><br><strong>Monitoring</strong> - Heroku is back online. JMWE is thus operational again, but we will continue to monitor the situation for now.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>12:10</var> EST</small><br><strong>Identified</strong> - Heroku, our hosting / platform provider, is currently experiencing a major outage. As a result, JMWE is currently down.<br />We are working with Heroku to get more information on their incident and restore our service as soon as possible.</p>tag:status.innovalog.com,2005:Incident/93970522022-02-24T06:28:33-05:002022-02-24T06:28:33-05:00Post-function and event-based actions execution are delayed<p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>06:28</var> EST</small><br><strong>Resolved</strong> - The problem is now resolved, and all post-functions and event-based actions that were delayed have now been processed.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>05:16</var> EST</small><br><strong>Identified</strong> - We have identified the cause of the problem and JMWE has now caught up with pending post-function and event-based action executions. <br />We are now investigating how to prevent this from happening again in the future.</p><p><small>Feb <var data-var='date'>24</var>, <var data-var='time'>04:49</var> EST</small><br><strong>Investigating</strong> - Post-function and event-based actions are currently running with a significant delay. We are investigating the root cause of this incident.</p>tag:status.innovalog.com,2005:Incident/82539232021-10-18T08:28:16-04:002021-10-18T08:28:16-04:00Post-function executions are delayed<p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>08:28</var> EDT</small><br><strong>Resolved</strong> - The incident is now resolved.<br />We have identified the root cause of the issue and put in place a safeguard so that it doesn't happen again.<br />We apologize for the inconvenience it might have caused.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>05:50</var> EDT</small><br><strong>Monitoring</strong> - The incident has now been resolved. All post-functions are running in a timely manner.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>04:45</var> EDT</small><br><strong>Investigating</strong> - Post-functions are currently running at a slow pace. We are investigating the root cause and will update this ticket as soon as we know more.</p>tag:status.innovalog.com,2005:Incident/59954782021-01-11T14:47:09-05:002021-01-11T14:47:09-05:00Post function execution delays<p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>14:47</var> EST</small><br><strong>Resolved</strong> - We have now verified that the fix works consistently.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>08:14</var> EST</small><br><strong>Monitoring</strong> - We have identified the issue and implemented a fix. Post functions are now running normally.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>05:50</var> EST</small><br><strong>Investigating</strong> - We are seeing an increased time to process post functions. We are currently investing this issue.</p>tag:status.innovalog.com,2005:Incident/53646222020-10-19T18:05:00-04:002020-10-19T18:05:00-04:00Post-function executions are delayed<p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>18:05</var> EDT</small><br><strong>Resolved</strong> - The incident is now over. All post-functions are running normally.<br />Sorry for the inconvenience this might have caused.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>17:55</var> EDT</small><br><strong>Monitoring</strong> - We have identified the problem and implemented a fix. Delayed post-functions are catching up, we expect everything to be back to normal in the next 5 minutes.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>17:31</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating an issue with post-function executions, which are currently delayed.</p>tag:status.innovalog.com,2005:Incident/49539812020-08-27T17:00:54-04:002020-08-27T17:00:54-04:00Delays in post-function exections<p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>17:00</var> EDT</small><br><strong>Resolved</strong> - The incident is now resolved. All post-functions are running normally.</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>16:17</var> EDT</small><br><strong>Monitoring</strong> - At 6:32pm UTC, post-function executions starting getting delayed. We have identified the root cause and implemented a fix. We are currently monitoring the solution, and post-functions that were delayed are expected to finish running by 8:20pm UTC. We apologize for the inconvenience.</p>tag:status.innovalog.com,2005:Incident/40485062020-05-06T14:41:22-04:002020-05-06T14:41:22-04:00JMWE Logs currently not available<p><small>May <var data-var='date'> 6</var>, <var data-var='time'>14:41</var> EDT</small><br><strong>Resolved</strong> - The incident has now been resolved by our logging infrastructure provider, and JMWE Logs are accessible again.</p><p><small>May <var data-var='date'> 6</var>, <var data-var='time'>14:34</var> EDT</small><br><strong>Identified</strong> - The issue has been identified by our logging infrastructure provider and a fix is being implemented by them</p><p><small>May <var data-var='date'> 6</var>, <var data-var='time'>14:08</var> EDT</small><br><strong>Investigating</strong> - JMWE Logs are currently unavailable, due to an incident at our logging infrastructure provider. They are aware of the issue and are currently investigating it.</p>tag:status.innovalog.com,2005:Incident/23038412019-04-03T23:03:42-04:002019-04-03T23:03:42-04:00Elevated errors while calling Jira's REST APIs, resulting in post-function failures<p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>23:03</var> EDT</small><br><strong>Resolved</strong> - Atlassian has identified a faulty server as the cause of these errors. The faulty hardware has been removed from their load balancer and everything should be back to normal.</p><p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>11:40</var> EDT</small><br><strong>Identified</strong> - Atlassian has identified the issue and is currently working on a resolution. They are tracking the problem as https://jira.atlassian.com/browse/JRACLOUD-71832 and https://jira.atlassian.com/browse/JRACLOUD-70291.</p><p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>15:34</var> EDT</small><br><strong>Investigating</strong> - Since Monday, April 1 at 12:30pm UTC, we are seeing a much-elevated number of connection errors to the Jira REST API, resulting in post-function failures.<br />We are currently investigating this with Atlassian.</p>tag:status.innovalog.com,2005:Incident/22443632019-03-07T11:09:26-05:002019-03-07T11:09:27-05:00Cannot access logs from the JMWE logs page<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>11:09</var> EST</small><br><strong>Resolved</strong> - The issue is now resolved.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>06:38</var> EST</small><br><strong>Identified</strong> - Our logging infrastructure provider is still working on a fix.<br />In the meantime, if you need to access your JMWE logs for troubleshooting purposes, please raise a support request at https://innovalog.atlassian.net/servicedesk/customer/portal/10 and we will assist you.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>10:55</var> EST</small><br><strong>Investigating</strong> - Our logging infrastructure provider is currently experiencing some technical difficulties, resulting in random failures when access logs from the JMWE logs page.<br />We are working with them to resolve the issue.</p>tag:status.innovalog.com,2005:Incident/21202392018-12-28T18:52:52-05:002018-12-28T18:52:52-05:00[Jira Issue] Some post-functions are not being fired up by Jira Cloud<p><small>Dec <var data-var='date'>28</var>, <var data-var='time'>18:52</var> EST</small><br><strong>Resolved</strong> - Atlassian had now resolved the issue. Post-functions should be running again.</p><p><small>Dec <var data-var='date'>28</var>, <var data-var='time'>17:15</var> EST</small><br><strong>Update</strong> - Atlassian has confirmed that there is a problem with web hooks / post functions in Jira Cloud affecting approximately 20% of requests. They are working to identify the root cause and get a fix out as soon as possible.</p><p><small>Dec <var data-var='date'>28</var>, <var data-var='time'>17:12</var> EST</small><br><strong>Identified</strong> - Atlassian has identified an issue with Jira Cloud Web hooks and post-functions, which are not triggering in approximately 20% of cases. For more information, see https://status.atlassian.com/incidents/dlv7w7xn7zpy</p>tag:status.innovalog.com,2005:Incident/20206612018-11-07T01:00:07-05:002018-11-07T01:08:33-05:00Post-functions not running immediately<p><small>Nov <var data-var='date'> 7</var>, <var data-var='time'>01:00</var> EST</small><br><strong>Resolved</strong> - On Nov. 6, 2018 at 21:48:14 UTC, some JMWE post-functions began to be delayed. This was caused by an incident with one of our platform providers.<br />Post-function executions were not lost, just delayed.<br />All delayed post-functions have now been run. We are continuing to monitor the situation.</p>tag:status.innovalog.com,2005:Incident/18770512018-08-24T11:35:59-04:002018-08-24T12:36:54-04:00Copy field value from linked issues (JMWE add-on) post-function failures<p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>11:35</var> EDT</small><br><strong>Resolved</strong> - We identified a bug that caused failures of the Copy field value from linked issues (JMWE add-on) post-function starting Aug. 23, 2018 at 17:09:30 UTC.<br />This bug has been fixed and the post-function is operational again since Aug. 24 at 17:34:14 UTC.<br />We apologize for the inconvenience this might have caused.</p>tag:status.innovalog.com,2005:Incident/15629832018-01-11T19:09:00-05:002018-01-12T04:18:16-05:00Logs currently show only a small subset of data<p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>19:09</var> EST</small><br><strong>Resolved</strong> - Logs are now available without any issues.
<br />However, logs for Jan 11, 5am UTC between 5pm UTC and 11pm UTC will not appear correctly. We apologize for the inconvenience.</p><p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>17:28</var> EST</small><br><strong>Investigating</strong> - We are currently experiencing a major issue with our logging infrastructure. Most of the log entries are not currently visible on the JMWE Logs page.
<br />We are working with our infrastructure provider to resolve the issue.</p>tag:status.innovalog.com,2005:Incident/15217212017-11-30T10:28:17-05:002017-11-30T10:28:17-05:00JMWE Logs page doesn't respond<p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>10:28</var> EST</small><br><strong>Resolved</strong> - Our logging infrastructure is back online, and the JMWE logs page works again.
<br />We will continue to monitor the service to make sure it stays that way.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>08:59</var> EST</small><br><strong>Investigating</strong> - Our logging infrastructure, run by a third-party provider, is unresponsive. We are working with them to get the issue resolved asap.</p>