Month: October 2024

Atlassian Analytics – Partial Data availability for some Atlassian Analytics customers

by

Oct 17, 07:10 UTC
Resolved – This incident has been resolved.

Oct 17, 07:08 UTC
Update – All impacted data for all customers is now successfully restored. Closing the incident communication.

Oct 16, 11:04 UTC
Update – Jira, Opsgenie, and JSM tables are fixed. We are continuing to work through a few remaining issues with data in Confluence tables for a small subset of customers. Next update in 12 hours or the the issue is fully resolved.

Oct 15, 22:34 UTC
Update – Jira, Opsgenie, and JSM tables are fixed. We are continuing to work through a few remaining issues with data in Confluence tables for a subset of customers. Next update in 12 hours or the the issue is fully resolved.

Oct 15, 13:06 UTC
Monitoring – We have remediated data issues for Jira, Opsgenie, and JSM tables. There is one outstanding issue with Confluence tables we are working to resolve. Next update in 12 hours or when the issue is fully resolved.

Oct 15, 00:08 UTC
Update – We are continuing to remediate data for impacted customers. Next update in 12 hours

Oct 14, 11:53 UTC
Update – Root cause of the issue identified and fixed. Data remediation for impacted customers is in progress. Next update in 12 hours.

Oct 14, 05:39 UTC
Investigating – Some Atlassian Analytics customers could experience partial data availability, due to a bug in data migrations across regions. Root cause is identified and we are actively working towards a fix as well as remediating impacted customer data.

OneLogin – OneLogin Engineers have identified an issue with Single Sign-On Services; some customers may be experiencing degraded performance. We are working to quickly restore service.

by
October 16, 2024 6:06AM PDT
Investigating – OneLogin Engineers have identified an issue with Single Sign-On Services; some customers may be experiencing degraded performance. We are working to quickly restore service.

October 16, 2024 7:06AM PDT
Investigating – Teams are investigating the ongoing issue, there is no current ETA for a resolution.

October 16, 2024 7:47AM PDT
Investigating – Our Team has started to remediate the situation and an increasing number of users are successfully accessing the portal page, other services are working as normal.

October 16, 2024 8:43AM PDT
Investigating – We continue to see improvements. Many users are successfully accessing the portal page and other services are working as normal.

October 16, 2024 9:48AM PDT
Investigating – We continue to see improvements. Many users are successfully accessing the portal page and other services are working as normal.

October 16, 2024 10:57AM PDT
Investigating – Our engineers continue to see improvements. Many users are successfully accessing the portal page and other services are working as normal.

October 16, 2024 11:29AM PDT
Investigating – Our engineers continue to see improvements. Many users are successfully accessing the portal page and other services are working as normal.

October 16, 2024 12:52PM PDT
Identified – Our engineering team has identified the issue and is implemented mitigation steps. We continue to see improvements for users accessing the portal and other services are working as normal.

October 16, 2024 1:44PM PDT
Identified – Our engineering team has identified the issue and is implemented mitigation steps. We continue to see improvements for users accessing the portal and other services are working as normal.

October 16, 2024 2:12PM PDT
Monitoring – Our engineering team has identified the issue and is completed mitigation steps. We continue to see improvements for users accessing the portal and other services are working as normal.

October 16, 2024 2:45PM PDT
Resolved – The incident is now resolved, engineers have mitigated the issue.

Trello – Boards are demonstrating slow performance for Workspace Guests

by

Oct 15, 18:25 EDT
Resolved – On October 15th, we identified that some boards were performing slowly for Workspace Guests. A fix has been released to all affected users, and the slow performance issue is fixed. No further impact has been observed.

Oct 15, 17:48 EDT
Monitoring – On October 15th, we identified that some boards were performing slowly for Workspace Guests.

We have taken action to mitigate this issue and we will continue to monitor for the next 30 minutes before marking the incident as resolved.

Oct 15, 17:08 EDT
Identified – We’ve identified the issue that caused the issue for slow performance for boards when loading them and opening cards for Workspace Guests, and we’re now working on a fix for the problem and will provide a new update here soon.

Oct 15, 12:58 EDT
Update – We’re continuing to investigating issues with boards that are experiencing slow performance when loading them and opening cards for Workspace Guests, and we will provide updates here soon

Oct 15, 10:56 EDT
Update – We are continuing to investigate issues with some large boards that are experiencing poor performance when loading them and opening cards, and will provide updates as we learn more.

Oct 14, 17:12 EDT
Update – We are continuing to investigate issues with some large boards that are experiencing poor performance when loading them and opening cards, and will provide updates as we learn more.

Oct 14, 15:23 EDT
Update – We are continuing to investigate issues with some large boards that are experiencing poor performance when loading them and opening cards, and will provide updates as we learn more.

Oct 14, 14:43 EDT
Investigating – We are investigating issues with some large boards that are experiencing poor performance when loading them and opening cards, and we will provide updates here soon.

OneLogin – OneLogin Engineers have identified an issue with Single Sign-On Services; customers may be experiencing a service disruption. We are working to quickly restore service.

by
October 15, 2024 5:12AM PDT
Investigating – OneLogin Engineers have identified an issue with Single Sign-On Services; some customers may be experiencing degraded performance. We are working to quickly restore service.

October 15, 2024 5:45AM PDT
Investigating – We continue to investigate the degraded performance which is impacting our Single Sign-On services.

October 15, 2024 6:15AM PDT
Investigating – We continue to investigate the service disruption which is impacting our Single Sign-On services.

October 15, 2024 6:51AM PDT
Investigating – Our engineers are continuing to investigate the service disruption impacting our Single Sign-On Services.

October 15, 2024 7:27AM PDT
Investigating – Engineers continue to investigate the service disruption with the highest priority.

October 15, 2024 7:57AM PDT
Investigating – Investigation continues with priority to bring our services back online.

October 15, 2024 8:35AM PDT
Investigating – Engineers are continuing to investigate the issue.

October 15, 2024 9:35AM PDT
Investigating – We continue to investigate the degraded performance which is impacting our Single Sign-On services.

October 15, 2024 10:07AM PDT
Investigating – Engineers are continuing to research the Single Sign-On Service disruption. We have every available resource working on this issue to resolve as quickly as possible.

October 15, 2024 10:41AM PDT
Investigating – We are continuing to investigate the service failure issues with priority. We currently do not have an ETA for resolution.

October 15, 2024 11:09AM PDT
Investigating – All One Login Engineers are working diligently to remediate the situation, we thank you for your patience

October 15, 2024 11:49AM PDT
Investigating – Work continues to resolve the current service issues. This is being treated with the utmost urgency.

October 15, 2024 12:33PM PDT
Investigating – We are continuing to investigate the issue, however at this time there is no ETA on a resolution.

October 15, 2024 1:09PM PDT
Identified – Our team has taken steps to mitigate the issue. Users should be seeing services return to normal and we will continue to monitor the situation.

October 15, 2024 1:54PM PDT
Monitoring – Services are now returning to normal, we will continue to monitor all services.

October 15, 2024 2:26PM PDT
Resolved – All services have been confirmed to be fully operational. Outage is now resolved.

Asana – Major outage

by

Oct 15, 13:30 UTC
Resolved – Our engineers have fixed the issue and Asana which cause global downtime for ~10 minutes and connectivity issues for a small subset of users for around an hour after that. Asana is now running normally for all our customers.

Reddit – Web performance degraded

by

Oct 14, 08:33 PDT
Resolved – This incident has been resolved.

Oct 14, 01:12 PDT
Investigating – We are currently experience a performance degradation of our service. We are working to fix the problem.

Pinterest – Search, Related Pins – Partial outage

by

Oct 13, 16:01 PDT
Resolved – We’ve fully resolved the issue and Pinterest Search, Related Pins should no longer be experiencing a partial outage. Thanks for your patience.

Oct 13, 12:17 PDT
Monitoring – Pinterest Search, Related Pins is experiencing a partial outage. We’ve implemented a solution to reduce the number of people affected and have partially recovered. We are monitoring for any additional problems and hope to have the issue fully resolved soon.

Oct 13, 11:50 PDT
Investigating – Pinterest Search, Related Pins is experiencing a partial outage. Our engineering team is working to identify the cause and implement a solution as quickly as possible. We’ll share another update shortly.

HubSpot – Some files are unavailable for some customers

by

Oct 11, 10:50 EDT
Resolved – Between 17:15 UTC on Oct 10, 2024 and 21:05 UTC on Oct 10, 2024, customers on EU1 portals experienced their HubSpot pages showing 409 errors and loading issues. The root cause has been resolved, and no data was lost.

Oct 10, 16:04 EDT
Investigating – Some files aren’t available right now. We’re investigating what’s causing this issue and will update this page when we have more information.

OneLogin – Onelogin Engineers have identified SAML issues which might be impacting certain customers

by
October 10, 2024 10:24AM PDT
Investigating – Onelogin Engineers have identified SAML issues which might be impacting certain customers

October 10, 2024 10:30AM PDT
Investigating – Engineers are currently working on identifying the issue

October 10, 2024 10:37AM PDT
Monitoring – Incident has been mitigated, engineers are currently monitoring for any changes.

October 10, 2024 11:08AM PDT
Resolved – Incident has been mitigated. Services have been restored.

Asana – Errors loading Asana

by

Oct 10, 02:52 UTC
Resolved – We have seen full resolution. We add a postmortem within 72 hours.

Oct 10, 02:48 UTC
Update – We are continuing to monitor for any further issues.

Oct 10, 02:36 UTC
Monitoring – We have reverted the change, see recovery, and are monitoring.

Oct 10, 02:13 UTC
Investigating – Users who are attempting to open Asana are seeing errors. We believe this is due to a deployment, and we are reverting this change.