Status Tracker v0.24
Jan 9, 00:58 UTC
Resolved – This incident has been resolved.
Jan 8, 23:27 UTC
Investigating – We are currently seeing increased npm network errors that are causing elevated errors for some builds. Our team is continuing to monitor the situation. You can watch for npm updates on their status page, here: https://status.npmjs.org/incidents/jlqm624klvs2
Jan 8, 16:11 PST
Resolved – This incident has been resolved.
Jan 8, 14:42 PST
Monitoring – A fix has been implemented and we are monitoring the results.
Jan 8, 13:04 PST
Identified – We have identified the issue is with a 3rd party infrastructure provider, who are currently engaged and working on a fix.
Jan 8, 10:02 PST
Investigating – We are currently investigating an issue where the desktop/web app may fail to launch and display a blank screen. Reloading the client (CTRL/CMD + R) may fix the issue in some instances.
Jan 8, 23:07 UTC
Resolved – This incident has been resolved.
Jan 8, 22:56 UTC
Monitoring – A fix has been implemented and we are monitoring the results.
Jan 8, 21:38 UTC
Investigating – We are seeing some intermittent error rates that could cause file access errors and errors loading the recent files page. We are actively investigating.
Jan 8, 22:00 UTC
Resolved – This incident has been resolved.
Jan 8, 18:05 UTC
Update – All error rates continue to be nominal, and things are looking stable. We’ll continue monitoring for any reoccurring issues.
Jan 8, 14:36 UTC
Monitoring – We’ve identified the issue and have pushed a fix. Error rates have returned to nominal levels, but we are continuing to monitor just in case.
Jan 8, 13:52 UTC
Update – We are continuing to investigate this issue.
Jan 8, 13:38 UTC
Investigating – We are currently investigating this issue.
Incident began at 2025-01-08 06:54 and ended at 2025-01-08 08:07 (all times are US/Pacific).
We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support
(All Times US/Pacific)
Incident Start: 8 January 2025 6:54
Incident End: 8 January 2025 8:07
Duration: 1 hour, 13 minutes
Affected Services and Features:
Regions/Zones: europe-west10, asia-south1, europe-west1, us-central1, asia-southeast2, us-east1, us-east5, asia-south2, us-south1, me-central1
Customers publishing from other regions may have also experienced the issue if the message storage policies [1] are set to store and process the messages in the above-mentioned regions.
Description:
Google Cloud Pub/Sub experienced a service outage in multiple regions for a duration of 1 hour and 13 minutes resulting in customers unable to publish or subscribe to the messages.
From preliminary analysis, the root cause of the issue was a configuration change which was rolled back to restore the service. Google will complete a full Incident Report in the following days that will provide a full root cause.
Customer Impact:
Google Cloud Pub/Sub : Customers were unable to publish or subscribe to the messages in the impacted regions. Publishing the messages from other regions may also have been impacted, if they have any of the impacted regions in their message storage policies. Backlog stats metric might be stale or missing.
Google BigQuery Data Transfer Service : Customers experienced failures with data transfers runs.
Cloud Logging : All Cloud Logs customers exporting logs to Cloud Pub/Sub experienced a delay in the log export for a duration of 26 minutes.
Reference(s):
[1] https://cloud.google.com/pubsub/docs/resource-location-restriction#message_storage_policy_overview
Affected products: Google Cloud Pub/Sub
Affected locations: Mumbai (asia-south1), Delhi (asia-south2), Jakarta (asia-southeast2), Belgium (europe-west1), Berlin (europe-west10), Doha (me-central1), Iowa (us-central1), South Carolina (us-east1), Columbus (us-east5), Dallas (us-south1)