Month: November 2024

Bubble – Issues with Main Bubble Cluster

by

Nov 27, 18:30 EST
Resolved – Our systems are functional and we are closing out this incident.

Nov 27, 18:29 EST
Investigating – We are investigating reports of issues with our systems.

Anthropic – Elevated errors for requests to Anthropic API

by

Nov 27, 10:57 PST
Resolved – This incident has been resolved.

Nov 27, 10:28 PST
Monitoring – A fix has been implemented and we are monitoring the results.

Nov 27, 09:39 PST
Investigating – We are currently investigating this issue.

Nov 27, 08:21 PST
Monitoring – A fix has been implemented and we are monitoring the results.

Nov 27, 07:29 PST
Investigating – We are currently investigating elevated errors on requests to Claude 3.5 Sonnet on the API, Claude.ai, and the Anthropic Console.

DigitalOcean – Network Connectivity in SFO3 Region

by

Nov 27, 12:03 UTC
Resolved – Our Engineering team identified and resolved the networking issue in our SFO3 region.

From 10.02 UTC to 11:44 UTC, users may have experienced connectivity issues, latency, and timeout errors while interacting with Droplet-based services and App Platform.

The impact has been mitigated and services should be working normally at this time.

If you continue to experience problems, please open a ticket with our support team. Thank you for your patience and we apologize for any inconvenience.

Nov 27, 11:44 UTC
Monitoring – Our engineering team has implemented a fix to resolve the networking issue in our SFO3 region. Users should no longer encounter connectivity issues, latency, and timeout errors while interacting with Droplet-based services and App Platform.

We will post an update as soon as the issue is fully resolved.

Nov 27, 11:31 UTC
Identified – As of 11:03 UTC, our Engineering team has identified the cause of the networking issue in our SFO3 region and is actively working on a fix. During this time users may experience connectivity issues, latency, and timeout errors while interacting with Droplet-based services and App Platform.

We will post an update as soon as additional information is available.

Nov 27, 11:03 UTC
Investigating – As of 10.02 UTC, our Engineering team is investigating an issue related to the SFO3 network maintenance here: https://status.digitalocean.com/incidents/rk0qtczt9j1c

During this time users may experience connectivity issues, latency, and timeout errors while interacting with Droplet-based services and App Platform.

We apologize for the inconvenience and will share an update once we have more information.

DigitalOcean – SFO3 Network Maintenance

by

Nov 27, 12:00 UTC
Completed – The scheduled maintenance has been completed.

Nov 27, 10:00 UTC
In progress – Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 25, 10:11 UTC
Scheduled – Start: 2024-11-27 10:00 UTC
End: 2024-11-27 14:00 UTC

During the above window, our Networking team will be performing maintenance on core switches in our SFO3 datacenter as part of network upgrades.

Expected Impact:

These upgrades are designed and tested to be seamless, and we do not expect any impact to customer traffic due to this maintenance. Should an unexpected issue arise, a possible outcome would be temporary loss of connectivity or increased latency for a brief period of time. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Supabase – Branching unavailable

by

Nov 27, 11:41 UTC
Resolved – This incident has been resolved.

Nov 27, 08:40 UTC
Monitoring – A fix has been implemented and we are monitoring the results.

Nov 27, 00:07 UTC
Identified – The issue has been identified and a fix is being implemented.

Nov 26, 18:01 UTC
Monitoring – A fix has been implemented and we are monitoring the results.

Nov 26, 15:16 UTC
Identified – The branching service is unavailable due to an upstream issue with fly.io (https://status.flyio.net/incidents/7lfnb87h43hf) impacting internal service connectivity and workload provisioning.

Nov 26, 09:25 UTC
Monitoring – Branching service was unavailable due to an upstream issue with fly.io (https://status.flyio.net/). The upstream issue has now been resolved and branching should be available once again.

Nov 26, 05:04 UTC
Investigating – We are currently investigating this issue.

Supabase – Compute capacity issues observed in the eu-west-3b availability zone

by

Nov 27, 09:58 UTC
Identified – We have identified an issue with our cloud provider regarding insufficient compute capacity in the eu-west-3b availability zone.
We have disabled the ability to restart projects, compute or database version upgrades within eu-west-3b.

Project provisioning in eu-west-3, as well as all operations for projects located in the eu-west-3a and eu-west-3c availability zones are available.

Nov 27, 09:57 UTC
Investigating – We are currently investigating this issue.

Anthropic – Elevated errors for requests to Claude 3.5 Sonnet

by

Nov 27, 01:17 PST
Resolved – This incident has been resolved.

Nov 27, 01:09 PST
Monitoring – A fix has been implemented and we are monitoring the results.

Nov 27, 00:53 PST
Investigating – We are currently investigating elevated errors on requests to Claude 3.5 Sonnet on the API, Claude.ai, and the Anthropic Console.

Jira – Microsoft API Failure Impacting Jira and JSM Email Processing

by

Nov 27, 06:06 UTC
Resolved – We are pleased to inform you that the recent issues affecting email processing for Microsoft accounts, which impacted our Jira and JSM Cloud customers, have been successfully resolved.

This resolution follows Microsoft’s reversion of the changes that initially caused the disruption. We appreciate your patience and understanding as we worked closely with Microsoft to address this matter promptly.

Nov 25, 15:32 UTC
Monitoring – Microsoft has identified the root cause of the API failures and provided an update that they are working on a solution. We are now monitoring this closely.

Nov 25, 10:51 UTC
Identified – We are investigating an issue with email processing that is impacting some Jira and JSM Cloud customers. Multiple Microsoft APIs are failing, which is causing emails not to be marked as read and creating duplicate tickets, comments, and customer notifications for email requests in JSM, as well as internal notifications in Jira. The impact started at 7:00 UTC. This is a widespread issue on Microsoft’s end, and we are currently awaiting their resolution

Rovo – 3P Search failing intermittently

by

Nov 27, 05:39 UTC
Resolved – The issue has been resolved and the service is operating normally.

Nov 19, 05:54 UTC
Investigating – We are actively exploring different approaches and continue to work on resolving the intermittent failures affecting some users in the 3P search for Rovo.

Nov 18, 22:03 UTC
Identified – We are actively working on resolving the issues with the 3P Search for Rovo. Our team is currently implementing a fix to alleviate the problem.

Nov 18, 20:36 UTC
Update – We are investigating intermittent errors with 3P Search that is impacting Rovo Cloud customers. We will provide more details within the next hour.

Nov 18, 20:34 UTC
Investigating – We are investigating intermittent errors while performing 3P search. We will provide more details once we identify the root cause.