Month: October 2024

Guard – Admin hub not responding

by

Nov 1, 05:22 UTC
Resolved – Between November 1st, 02:57 AM UTC, and November 1st, 05:03 AM UTC, we experienced an issue with the Atlassian Admin hub not responding to some users. The issue has been resolved, and the service is operating normally.

Nov 1, 05:03 UTC
Monitoring – We have identified the root cause of the Atlassian Admin hub not responding and have mitigated the problem. We are now monitoring closely.

Nov 1, 05:01 UTC
Identified – We continue to work on resolving the issue with the Atlassian admin hub not responding. We have identified the root cause and expect recovery shortly.

Nov 1, 04:37 UTC
Investigating – We are investigating reports of intermittent errors when logging into the Atlassian admin hub at https://admin.atlassian.com.

Once we identify the root cause, we will provide more details.

New Relic – Data Processing and Alerts Delays in EU Region

by

Nov 1, 03:11 UTC
Resolved – Between 00:36 UTC and 02:35 UTC on November 1st, 2024, some customers in the EU region may have experienced data visibility irregularities and delayed alert notifications for affected data.

The interruption with our cloud provider has been resolved and affected services have returned to normal operations.

Nov 1, 02:34 UTC
Monitoring – We are seeing signs of recovery and will continue to monitor affected services.

Nov 1, 02:09 UTC
Identified – Our cloud provider has confirmed an interruption and is working to resolve the issue impacting New Relic services.

Nov 1, 01:41 UTC
Update – We are continuing to investigate this interruption with our cloud provider.

Nov 1, 01:09 UTC
Investigating – Some customers in the EU region may be experiencing data visibility irregularities and delayed alert notifications for affected data.

We are investigating this interruption and have escalated to our cloud provider.

New Relic – Delayed alert notifications for US Region

by

Oct 31, 07:15 UTC
Resolved – Between 07:03 UTC and 07:15 UTC on October 31st, some customers in the US region may have experienced delayed or missing alert notifications. This issue has been resolved and impacted services have returned to normal operations.

HubSpot – Some HubSpot updates are currently delayed. We’re investigating the cause of this issue.

by

Oct 30, 15:01 EDT
Resolved – Starting 12:04 PM EDT (UTC -04:00) on Oct 30, 2024, customers experienced update delays in several tools. We’ve addressed the issue that was causing the delays. No data has been lost. We appreciate your patience as we resolved the issue.

Oct 30, 13:27 EDT
Monitoring – We’ve addressed the issue that was causing update delays starting at 12:04 PM EDT (UTC -04:00) on Oct 30, 2024. We’re monitoring the tools closely to ensure all processes recover properly.

Oct 30, 12:45 EDT
Investigating – Some HubSpot updates are currently delayed. We’re investigating the cause of this issue.

Atlassian Developer – Forge Create Fails with Forge CLI v10.8

by

Oct 30, 18:30 UTC
Resolved – At 2024-10-29 10:00 UTC users using the latest Forge CLI v10.8.0 would be facing issues including creating new Forge apps. Developers would receive the error:
Error: The “path” argument must be of type string. Received undefined
Users who installed the latest version need to downgrade the CLI to version 10.7.0 using npm install -g @forge/[email protected] for anyone who is on 10.8.0 version currently. You can know the current version you are in by executing the forge –version command
The issue was mitigated at 2024-10-30 5:20 UTC where users using the latest Forge CLI ( npm install -g @forge/cli@latest) will automatically point to 10.7.0
The team apologies for inconvenience caused.
There will be no further communications for this incident.

Pinterest – Ads Manager — Creation, Ads Manager — General

by

Oct 29, 15:16 PDT
Resolved – We’ve fully resolved the issue and Pinterest Ads Manager — Creation, Ads Manager — General should be working as expected. Thanks for your patience.

Oct 29, 14:25 PDT
Update – Pinterest Ads Manager — Creation, Ads Manager — General is experiencing a partial outage. Our engineering team is working to identify the cause and implement a solution as quickly as possible. We’ll share another update shortly.

Oct 29, 14:22 PDT
Investigating – Pinterest Ads Manager — General is experiencing a partial outage. Our engineering team is working to identify the cause and implement a solution as quickly as possible. We’ll share another update shortly.

Okta – Resolved Service Degradation

by

Okta has received reports of SMS delivery issues affecting users in China. Okta’s Engineering team is actively investigating. At this time, we recommend that customers in China use a secondary MFA option if available.

Atlassian Developer – App deployment degradation

by

Oct 28, 22:41 UTC
Resolved – This issue is now resolved and performance is back to normal.
We recommend any apps deployed between 9:24 UTC on 2024-10-23
05:56 UTC 2024-10-25 to be re-deployed in case the app users identify any latency issues.

Oct 23, 09:53 UTC
Monitoring – Between 08:59 UTC and 9:24 UTC on 2024-10-23 we have observed increased error rate on app deployments, similar to problem that happened on 2024-10-24.

After investigation we found an condition in one of cells in EU region which was causing deployments to fail.

We applied change to migrate new deployments off the impacted EU cell which mitigated impact.

We are keeping this incident in monitoring state and will resolve it once underlying issue is resolved and deployments are resumed to the affected cell.

From perspective of App developers this means that deployments of apps (new apps and updates to existing ones) will work correctly but may introduce minor performance degradation due to function being invoked from nearest EU cell when customer is located in the affected cell. Because of this we will post an update advising to re-deploy affected apps once this incident is resolved.

Discord – Message search indexing behind

by

Oct 28, 02:39 PDT
Resolved – This incident has been resolved.

Oct 27, 17:25 PDT
Monitoring – Recovery of the backlog of messages indexed for search is happening

Oct 27, 14:14 PDT
Investigating – We are currently investigating an issue with message search indexing falling behind