Day: January 7, 2025

DigitalOcean – Network Connectivity in SFO3 Region

by

Jan 7, 16:43 UTC
Resolved – From 13:09 – 14:48 UTC, we experienced an issue with IPv6 network connectivity in our SFO3 region. Our Engineering team was able to identify the root cause, revert changes, and resolve the issue.

During that time, connectivity via IPv6 to services in SFO3 was impacted. This included, in particular, connectivity to and provisioning of Load Balancers, control plane connectivity to Kubernetes clusters, and connectivity to App Platform Apps.

If you continue to experience problems, please open a ticket with our support team. Thank you for your patience and we apologize for any inconvenience.

Jan 7, 15:51 UTC
Monitoring – Our engineering team has implemented a fix to resolve the networking issue in our SFO3 region. Users should no longer encounter connectivity issues, provisioning Load Balancers, latency, and timeout errors while interacting with resources in the SFO3 region.

We will post an update as soon as the issue is fully resolved.

Jan 7, 14:47 UTC
Update – Our Engineering team is continuing to investigate the cause of the issue with networking in our SFO3 region and is taking action to mitigate this incident by reverting recent changes. In addition to connectivity issues, users may see failures with provisioning Load Balancers in SFO3.

We are seeing some services recover, but expect to see a few more short periods of network connectivity issues as the reverts progress.

We will continue to share updates as soon as we have new information.

Jan 7, 13:35 UTC
Investigating – Our engineering team is investigating an issue related to the SFO3 network maintenance here: https://status.digitalocean.com/incidents/lf0njg8ysj0w

During this time users may experience connectivity issues, latency, and timeout errors while interacting with resources in the SFO3 data center.

We apologize for the inconvenience and will share an update once we have more information.

GitHub – Incident with Actions resulting in degraded performance

by

Jan 7, 16:39 UTC
Resolved – This incident has been resolved.

Jan 7, 16:39 UTC
Update – Issues is operating normally.

Jan 7, 16:38 UTC
Update – Actions is operating normally.

Jan 7, 16:38 UTC
Update – Webhooks is operating normally.

Jan 7, 16:09 UTC
Update – Webhooks is experiencing degraded performance. We are continuing to investigate.

Jan 7, 15:59 UTC
Update – We have identified a configuration issue that we believe is the source of the Action workflow job delays and page latency increases. We are continuing to investigate and mitigate the issue.

Jan 7, 15:51 UTC
Update – Issues is experiencing degraded performance. We are continuing to investigate.

Jan 7, 15:17 UTC
Update – Users may see delays with Action workflow jobs in the UI and API responses. Additionally, several endpoints, including some Pull Request pages are experiencing increased latency. We are continuing to investigate the issue.

Jan 7, 14:49 UTC
Investigating – We are investigating reports of degraded performance for Actions

DigitalOcean – Error adding card details

by

Jan 7, 14:14 UTC
Resolved – Our engineering team has resolved the issue preventing some users from adding new credit cards from the control panel. If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Jan 7, 13:38 UTC
Monitoring – Our engineering team has implemented a fix to resolve issues that some users experienced when attempting to add new credit cards to the Cloud Control Panel.

We are monitoring the situation and will post an update as soon as the issue is fully resolved.

Jan 7, 13:24 UTC
Investigating – Our engineering team is investigating reports of issues occurring when users attempt to add a new credit card in the billing settings page of the Cloud Control Panel.

During this time, some users may be unable to input card information. We apologize for the inconvenience and will share an update once we have more information.

Klaviyo – Delays in some review requests

by

Jan 7, 09:08 EST
Update – Our team has processed 94% of the delayed Ready to Review events. We will update this incident when the remaining 6% have been processed.

Dec 31, 08:50 EST
Update – Our engineering team is working to release the delayed Ready to Review events. We will continue to keep this incident up to date with our progress.

Dec 30, 12:48 EST
Identified – On December 9th at around 10:30 Eastern Time, we experienced a technical issue which impacted Ready to Review events. As a result, some Ready to Review events corresponding to orders that occurred between December 9th and December 27th are delayed. This could lead to delayed review requests being sent out via Flows.

Ready to Review events corresponding to orders that occurred after December 27th at around 2:15 Eastern Time are not delayed.

Our team has identified the issue and is working to release the backlogged Ready to Review events. We will keep this incident up to date with our progress.

DigitalOcean – SFO3 Network Maintenance

by

Jan 7, 14:00 UTC
Completed – The scheduled maintenance has been completed.

Jan 7, 10:00 UTC
In progress – Scheduled maintenance is currently in progress. We will provide updates as necessary.

Jan 5, 10:15 UTC
Scheduled – Start: 2025-01-07 10:00 UTC
End: 2025-01-07 14:00 UTC

During the above window, our Networking team will be making changes to core networking infrastructure, to improve performance and scalability in the SFO3 region.

Expected impact:

These upgrades are designed and tested to be seamless and we do not expect any impact to customer traffic due to this maintenance. If an unexpected issue arises, affected Droplets and Droplet-based services may experience increased latency or a brief disruption in network traffic. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket