DigitalOcean

Nov 29, 05:35 UTC
Resolved – Our Engineering team has confirmed the full resolution of this issue. If you continue to experience problems, please open a ticket with our support team. Thank you for being so patient, and we apologize for any inconvenience.

Nov 29, 04:24 UTC
Monitoring – Our domain registrar has taken action and the digitaloceanspaces.com domain is now recovering. We have verified the domain is resolving and are seeing error rates dropping for all impacted services. Users should begin seeing services returning to normal operation.

Our Engineering teams are monitoring the situation to ensure complete recovery and we will post a final update soon.

Nov 29, 03:47 UTC
Update – Our Engineering team has identified additional downstream services seeing elevated failure rates and broken functionality. At this time, the following services are impacted:

– Spaces object accessibility in all regions with objects hosted under digitaloceanspaces.com.

– Requests to create, push, or pull to/from Container Registries

– Backups for the Mongo Managed Databases product are not successfully uploading, so users attempting to use those backups will see errors.

– Creates and resizes for Load Balancers in all regions

– New builds for App Platform in all regions

– Create or updating Functions in all regions

We are in active contact with our vendor to resolve this as quickly as possible. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved.

We will continue to provide updates as we have new information.

Nov 29, 03:36 UTC
Update – Our Engineering team has identified additional downstream services seeing elevated failure rates and broken functionality. At this time, the following services are impacted:

– Spaces object accessibility in all regions with objects hosted under digitaloceanspaces.com.

– Requests to create, push, or pull to/from Container Registries

– Backups for the Mongo Managed Databases product are not successfully uploading, so users attempting to use those backups will see errors.

– Creates and resizes for Load Balancers in all regions

– New builds for the App Platform in all regions

We are in active contact with our vendor to resolve this as quickly as possible. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved.

We will continue to provide updates as we have new information.

Nov 29, 03:04 UTC
Update – Spaces accessibility in all regions continues to be impacted and users will experience errors when attempting to interact with objects hosted under digitaloceanspaces.com. Container Registries are also impacted and users will experience errors when attempting to create, push, or pull containers. Backups for the Mongo Managed Databases product are also not successfully uploading, so users attempting to use those backups will see errors.

We are in active contact with our vendor to resolve this as quickly as possible. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved.

We will continue to provide updates as we have new information.

Nov 29, 02:38 UTC
Update – Spaces accessibility in all regions continues to be impacted and users will experience errors when attempting to interact with objects hosted under digitaloceanspaces.com. Backups for the Mongo Managed Databases product are also not successfully uploading, so users attempting to use those backups will see errors.

We have engaged with our upstream vendor to resolve this incident as quickly as possible and are awaiting a further update. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved.

We understand these are critical issues for our customers and are working as quickly as possible to resolve this.

Nov 29, 01:50 UTC
Update – Our Engineering team has successfully deployed a solution to bring the Cloud Control Panel back online. Please refresh the page a few times to clear your cache.

However, we are still experiencing issues with Spaces. We are engaging with our upstream vendor to resolve this incident as quickly as possible. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved. We understand this is a critical issue for our customers and are working as quickly as possible to resolve this.

Nov 29, 01:31 UTC
Identified – Our Engineering team has identified this as a domain registrar issue for the entire digitaloceanspaces.com domain. We are engaging with our upstream vendor to resolve this incident as quickly as possible. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved. We understand this is a critical issue for our customers.

Our Cloud Control Panel is dependent on our Spaces product and is also experiencing an outage due to this issue.

Due to the Cloud Control Panel being down, if users need to submit a support ticket, they may do so here: https://www.digitalocean.com/company/contact/support

Nov 29, 01:10 UTC
Update – Our Engineering team is continuing to investigate this issue and all possible paths to remediation, as well as confirm if this issue is related to an upstream provider outage. We understand this is a critical issue for our customers.

Due to the Cloud Control Panel being down, if users need to submit a support ticket, they may do so here: https://www.digitalocean.com/company/contact/support

Nov 29, 00:34 UTC
Update – We are continuing to investigate this issue.

Nov 29, 00:21 UTC
Investigating – Our Engineering team is currently investigating an issue affecting portions of the Cloud Panel across all regions. Additionally, some Space Buckets may be inaccessible via the CDN. Users may experience difficulties accessing the cloud control panel, encounter DNS-related problems, or face interruptions when managing or accessing Spaces and Spaces CDN through.

We apologize for the inconvenience and will share an update once we have more information.

Nov 27, 12:03 UTC
Resolved – Our Engineering team identified and resolved the networking issue in our SFO3 region.

From 10.02 UTC to 11:44 UTC, users may have experienced connectivity issues, latency, and timeout errors while interacting with Droplet-based services and App Platform.

The impact has been mitigated and services should be working normally at this time.

If you continue to experience problems, please open a ticket with our support team. Thank you for your patience and we apologize for any inconvenience.

Nov 27, 11:44 UTC
Monitoring – Our engineering team has implemented a fix to resolve the networking issue in our SFO3 region. Users should no longer encounter connectivity issues, latency, and timeout errors while interacting with Droplet-based services and App Platform.

We will post an update as soon as the issue is fully resolved.

Nov 27, 11:31 UTC
Identified – As of 11:03 UTC, our Engineering team has identified the cause of the networking issue in our SFO3 region and is actively working on a fix. During this time users may experience connectivity issues, latency, and timeout errors while interacting with Droplet-based services and App Platform.

We will post an update as soon as additional information is available.

Nov 27, 11:03 UTC
Investigating – As of 10.02 UTC, our Engineering team is investigating an issue related to the SFO3 network maintenance here: https://status.digitalocean.com/incidents/rk0qtczt9j1c

During this time users may experience connectivity issues, latency, and timeout errors while interacting with Droplet-based services and App Platform.

We apologize for the inconvenience and will share an update once we have more information.

Nov 27, 12:00 UTC
Completed – The scheduled maintenance has been completed.

Nov 27, 10:00 UTC
In progress – Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 25, 10:11 UTC
Scheduled – Start: 2024-11-27 10:00 UTC
End: 2024-11-27 14:00 UTC

During the above window, our Networking team will be performing maintenance on core switches in our SFO3 datacenter as part of network upgrades.

Expected Impact:

These upgrades are designed and tested to be seamless, and we do not expect any impact to customer traffic due to this maintenance. Should an unexpected issue arise, a possible outcome would be temporary loss of connectivity or increased latency for a brief period of time. We will endeavor to keep any such impact to a minimum.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Nov 27, 03:00 UTC
Completed – The scheduled maintenance has been completed.

Nov 27, 01:52 UTC
Update – Our Engineering team has decided to extend the maintenance window by an hour to monitor the situation.

Maintenance is still underway and we will update once the maintenance is fully completed.

We sincerely apologize for any inconvenience this may have caused and appreciate your understanding.

Nov 27, 00:01 UTC
Update – From 22:25 – 23:15 UTC, all event-related actions including creating or deleting resources, were temporarily disabled in NYC3.

Users attempting any event-based operations across products hosted in NYC3 would have encountered error messages. While we anticipated event failures and delays, this impact period was longer than expected. We are taking steps to ensure this does not happen in future maintenances.

As of now, maintenance is still underway and we will update once the maintenance is fully completed.

We sincerely apologize for any inconvenience this may have caused and appreciate your understanding.

Nov 26, 22:00 UTC
In progress – Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 24, 22:03 UTC
Scheduled – Start: 2024-11-26 22:00 UTC
End: 2024-11-27 02:00 UTC

During the above window, our Networking team will be making changes to core networking infrastructure, to improve performance and scalability in the NYC3 region.

Expected impact:

During the maintenance window users may experience delays or failures with event processing for a brief duration on Droplets and Droplet-based services including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Nov 25, 22:05 UTC
Resolved – Our Engineering team is investigating an issue related to our ongoing SFO3 maintenance here: https://status.digitalocean.com/incidents/4kj7krrpyg3k

From 20:23 – 20:25 UTC, some services were impacted by a drop in networking. During that time, some Managed Kubernetes clusters experienced errors from the Kubernetes API and/or an increase in 5xx errors. Communication between other services and Block Storage Volumes may have been impacted as well.

The impact has been mitigated and services should be working normally at this time.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.