Status Tracker v0.23
Incident began at 2024-10-03 00:00 and ended at 2024-11-25 23:00 (all times are US/Pacific).
As of 11/25, our fix has rolled out to 100% of the affected registrations. Topic fanouts to affected registrations should now be succeeding.
Affected products: Cloud Messaging
Nov 29, 03:02 PST
Resolved – After further monitoring, this incident is now considered resolved. Box Notes Service has been restored to full functionality. If you continue to experience any issues, please contact Box Support at https://support.box.com.
Nov 29, 02:42 PST
Monitoring – Our team has taken steps to remediate this issue and is seeing improvement on Box Notes service. We are continuing to monitor for any additional impact.
Nov 29, 02:30 PST
Update – Our team has identified the underlying cause of this issue and is working to take remediating steps. We will provide additional updates as they become available.
Nov 29, 02:28 PST
Investigating – Our team is investigating an issue with Box Notes. Users may see errors or slowness when attempting to create or edit Box Notes. We will provide additional information as it becomes available.
Nov 29, 01:42 PST
Resolved – Our team has deployed a fix, and real time edits are working as expected now.
Nov 29, 01:00 PST
Investigating – Starting 11:30 PM PST, we started experiencing issues with realtime editing on multiplayer workspaces. Our engineering team is notified about it and they are working to resolve the issue. We will provide more updates as and when they become available.
Nov 29, 05:35 UTC
Resolved – Our Engineering team has confirmed the full resolution of this issue. If you continue to experience problems, please open a ticket with our support team. Thank you for being so patient, and we apologize for any inconvenience.
Nov 29, 04:24 UTC
Monitoring – Our domain registrar has taken action and the digitaloceanspaces.com domain is now recovering. We have verified the domain is resolving and are seeing error rates dropping for all impacted services. Users should begin seeing services returning to normal operation.
Our Engineering teams are monitoring the situation to ensure complete recovery and we will post a final update soon.
Nov 29, 03:47 UTC
Update – Our Engineering team has identified additional downstream services seeing elevated failure rates and broken functionality. At this time, the following services are impacted:
– Spaces object accessibility in all regions with objects hosted under digitaloceanspaces.com.
– Requests to create, push, or pull to/from Container Registries
– Backups for the Mongo Managed Databases product are not successfully uploading, so users attempting to use those backups will see errors.
– Creates and resizes for Load Balancers in all regions
– New builds for App Platform in all regions
– Create or updating Functions in all regions
We are in active contact with our vendor to resolve this as quickly as possible. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved.
We will continue to provide updates as we have new information.
Nov 29, 03:36 UTC
Update – Our Engineering team has identified additional downstream services seeing elevated failure rates and broken functionality. At this time, the following services are impacted:
– Spaces object accessibility in all regions with objects hosted under digitaloceanspaces.com.
– Requests to create, push, or pull to/from Container Registries
– Backups for the Mongo Managed Databases product are not successfully uploading, so users attempting to use those backups will see errors.
– Creates and resizes for Load Balancers in all regions
– New builds for the App Platform in all regions
We are in active contact with our vendor to resolve this as quickly as possible. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved.
We will continue to provide updates as we have new information.
Nov 29, 03:04 UTC
Update – Spaces accessibility in all regions continues to be impacted and users will experience errors when attempting to interact with objects hosted under digitaloceanspaces.com. Container Registries are also impacted and users will experience errors when attempting to create, push, or pull containers. Backups for the Mongo Managed Databases product are also not successfully uploading, so users attempting to use those backups will see errors.
We are in active contact with our vendor to resolve this as quickly as possible. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved.
We will continue to provide updates as we have new information.
Nov 29, 02:38 UTC
Update – Spaces accessibility in all regions continues to be impacted and users will experience errors when attempting to interact with objects hosted under digitaloceanspaces.com. Backups for the Mongo Managed Databases product are also not successfully uploading, so users attempting to use those backups will see errors.
We have engaged with our upstream vendor to resolve this incident as quickly as possible and are awaiting a further update. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved.
We understand these are critical issues for our customers and are working as quickly as possible to resolve this.
Nov 29, 01:50 UTC
Update – Our Engineering team has successfully deployed a solution to bring the Cloud Control Panel back online. Please refresh the page a few times to clear your cache.
However, we are still experiencing issues with Spaces. We are engaging with our upstream vendor to resolve this incident as quickly as possible. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved. We understand this is a critical issue for our customers and are working as quickly as possible to resolve this.
Nov 29, 01:31 UTC
Identified – Our Engineering team has identified this as a domain registrar issue for the entire digitaloceanspaces.com domain. We are engaging with our upstream vendor to resolve this incident as quickly as possible. Any DNS record under digitaloceanspaces.com will continue to fail until this incident is resolved. We understand this is a critical issue for our customers.
Our Cloud Control Panel is dependent on our Spaces product and is also experiencing an outage due to this issue.
Due to the Cloud Control Panel being down, if users need to submit a support ticket, they may do so here: https://www.digitalocean.com/company/contact/support
Nov 29, 01:10 UTC
Update – Our Engineering team is continuing to investigate this issue and all possible paths to remediation, as well as confirm if this issue is related to an upstream provider outage. We understand this is a critical issue for our customers.
Due to the Cloud Control Panel being down, if users need to submit a support ticket, they may do so here: https://www.digitalocean.com/company/contact/support
Nov 29, 00:34 UTC
Update – We are continuing to investigate this issue.
Nov 29, 00:21 UTC
Investigating – Our Engineering team is currently investigating an issue affecting portions of the Cloud Panel across all regions. Additionally, some Space Buckets may be inaccessible via the CDN. Users may experience difficulties accessing the cloud control panel, encounter DNS-related problems, or face interruptions when managing or accessing Spaces and Spaces CDN through.
We apologize for the inconvenience and will share an update once we have more information.
Nov 28, 16:47 PST
Resolved – This incident has been resolved.
Nov 28, 12:02 PST
Monitoring – A fix has been implemented and we are monitoring the results.
Nov 28, 08:23 PST
Resolved – This incident has been resolved.
Nov 27, 23:39 PST
Monitoring – A fix has been implemented and we are monitoring the results.
Nov 27, 22:35 PST
Investigating – We are currently troubleshooting an issue impacting specific API workloads.
Nov 28, 11:09 EST
Resolved – Our systems are functional and we are closing out this incident.
Nov 28, 11:08 EST
Investigating – We are investigating reports of issues with our systems.
Incident began at 2024-11-28 01:40 and ended at 2024-11-28 04:52 (all times are US/Pacific).
The issue with Vertex AI AutoML Image has been resolved for all affected users as of Thursday, 2024-11-28 04:48 US/Pacific.
We thank you for your patience while we worked on resolving the issue.
Affected products: Cloud Machine Learning, Vertex AI AutoML Image
Affected locations: Netherlands (europe-west4), Iowa (us-central1)
Nov 28, 02:42 PST
Resolved – This incident has been resolved.
Nov 28, 02:29 PST
Monitoring – We have resolved the issue with Zoom billing portal. We will continue to monitor and provide updates as we have them.
Nov 28, 01:55 PST
Identified – Our monitoring has detected issues with Zoom billing portal, we have identified and investigating the issue. We will provide updates as we have them.
November 28, 2024 10:39 UTC
Resolved – Functionality has been restored to the Customer Portal (customers.gitlab.com). The billing pages in GitLab.com are also available.