Day: November 26, 2024

Jira – Microsoft API Failure Impacting Jira and JSM Email Processing

by

Nov 27, 06:06 UTC
Resolved – We are pleased to inform you that the recent issues affecting email processing for Microsoft accounts, which impacted our Jira and JSM Cloud customers, have been successfully resolved.

This resolution follows Microsoft’s reversion of the changes that initially caused the disruption. We appreciate your patience and understanding as we worked closely with Microsoft to address this matter promptly.

Nov 25, 15:32 UTC
Monitoring – Microsoft has identified the root cause of the API failures and provided an update that they are working on a solution. We are now monitoring this closely.

Nov 25, 10:51 UTC
Identified – We are investigating an issue with email processing that is impacting some Jira and JSM Cloud customers. Multiple Microsoft APIs are failing, which is causing emails not to be marked as read and creating duplicate tickets, comments, and customer notifications for email requests in JSM, as well as internal notifications in Jira. The impact started at 7:00 UTC. This is a widespread issue on Microsoft’s end, and we are currently awaiting their resolution

Rovo – 3P Search failing intermittently

by

Nov 27, 05:39 UTC
Resolved – The issue has been resolved and the service is operating normally.

Nov 19, 05:54 UTC
Investigating – We are actively exploring different approaches and continue to work on resolving the intermittent failures affecting some users in the 3P search for Rovo.

Nov 18, 22:03 UTC
Identified – We are actively working on resolving the issues with the 3P Search for Rovo. Our team is currently implementing a fix to alleviate the problem.

Nov 18, 20:36 UTC
Update – We are investigating intermittent errors with 3P Search that is impacting Rovo Cloud customers. We will provide more details within the next hour.

Nov 18, 20:34 UTC
Investigating – We are investigating intermittent errors while performing 3P search. We will provide more details once we identify the root cause.

DigitalOcean – NYC3 Network Maintenance

by

Nov 27, 03:00 UTC
Completed – The scheduled maintenance has been completed.

Nov 27, 01:52 UTC
Update – Our Engineering team has decided to extend the maintenance window by an hour to monitor the situation.

Maintenance is still underway and we will update once the maintenance is fully completed.

We sincerely apologize for any inconvenience this may have caused and appreciate your understanding.

Nov 27, 00:01 UTC
Update – From 22:25 – 23:15 UTC, all event-related actions including creating or deleting resources, were temporarily disabled in NYC3.

Users attempting any event-based operations across products hosted in NYC3 would have encountered error messages. While we anticipated event failures and delays, this impact period was longer than expected. We are taking steps to ensure this does not happen in future maintenances.

As of now, maintenance is still underway and we will update once the maintenance is fully completed.

We sincerely apologize for any inconvenience this may have caused and appreciate your understanding.

Nov 26, 22:00 UTC
In progress – Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 24, 22:03 UTC
Scheduled – Start: 2024-11-26 22:00 UTC
End: 2024-11-27 02:00 UTC

During the above window, our Networking team will be making changes to core networking infrastructure, to improve performance and scalability in the NYC3 region.

Expected impact:

During the maintenance window users may experience delays or failures with event processing for a brief duration on Droplets and Droplet-based services including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Canva – Users are unable to use autofill to create designs using Enterprise Integrations.

by

Nov 27, 13:59 AEDT
Resolved – This incident is now resolved. Thank you for your patience and understanding.

Nov 27, 13:25 AEDT
Identified – – Jobs created with Autofill are currently unable to complete.
– None of the other Connect API features appears to be affected.
– Canva’s Editor and Homepage are not affected by this issue.
– We have identified the cause are working to mitigate.

GitLab – Duplicate Merge Request Events

by
November 26, 2024 18:43 UTC
Identified – Users may see duplicated events on merge requests, or failed merge attempts and merge request updates.

Please see https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18904 for more information.

November 26, 2024 19:04 UTC
Monitoring – A fix has been rolled out and merge request events should no longer be duplicated.

See the production issue for more details: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18904

November 26, 2024 19:52 UTC
Resolved – We have confirmed that duplicate merge request events have stopped being created.

Please see the production issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18904

OpenAI – Elevated Errors in API

by

Nov 26, 11:30 PST
Resolved – We experienced a brief period of elevated errors for API requests between approximately 10:45 AM – 11:15 AM PT.

Anthropic – Elevated errors on Claude 3.5 Sonnet

by

Nov 26, 10:12 PST
Resolved – This incident has been resolved.

Nov 26, 08:47 PST
Monitoring – A fix has been implemented and we are monitoring the results.

Nov 26, 06:17 PST
Investigating – We are currently investigating this issue.

Supabase – Compute capacity issues observed in the eu-west-3b availability zone

by

Nov 26, 18:06 UTC
Resolved – This incident has been resolved.

Nov 23, 22:14 UTC
Identified – We have identified an issue with our cloud provider regarding insufficient compute capacity in the eu-west-3b availability zone.
We have disabled the ability to restart projects, compute or database version upgrades within eu-west-3b.

Project provisioning in eu-west-3, as well as all operations for projects located in the eu-west-3a and eu-west-3c availability zones are available.

Nov 23, 22:10 UTC
Investigating – We are currently investigating this issue.

Vercel – Delays Issuing New SSL Certificates

by

Nov 26, 12:00 UTC
Resolved – This incident has been resolved.

Nov 26, 11:39 UTC
Monitoring – A fix has been implemented and we are monitoring the backlog of delayed certificates, which is decreasing.
Once the backlog of the affected domains is clear we will mark the incident as resolved

Nov 26, 11:24 UTC
Identified – The issue has been identified and a fix is being implemented.

Nov 26, 11:12 UTC
Investigating – We are currently investigating an issue causing new SSL certificates to be delayed.