Status Tracker v0.23
Incident began at 2024-12-19 11:31 and ended at 2024-12-19 17:30 (all times are US/Pacific).
The problem with Vertex AI In Firebase has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.
Affected products: Vertex AI for Firebase
Incident began at 2024-12-19 11:36 and ended at 2024-12-19 17:35 (all times are US/Pacific).
The issue with Vertex Gemini API has been resolved for all affected users as of Thursday, 2024-12-19 17:30 US/Pacific.
We thank you for your patience while we worked on resolving the issue.
Affected products: Vertex Gemini API
Affected locations: Iowa (us-central1)
Incident began at 2024-12-19 08:25 and ended at 2024-12-19 11:05 (all times are US/Pacific).
We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support .
(All Times US/Pacific)
Incident Start: 19 December, 2024 08:15
Incident End: 19 December, 2024 10:55
Duration: 2 hours, 40 minutes
Affected Services and Features:
Vertex Gemini API
Regions/Zones: multi-region:asia
Description:
Google Vertex Gemini API customers sending traffic to gemini models in multi-region:asia experienced an increase in 5xx errors up to 100% for a duration of 2 hours, 40 minutes. From preliminary analysis, the root cause of the incident is stemmed from a Vertex service dependency ending up in a bad state due to a configuration process.
The situation was mitigated by increasing resources, enabling successful autopilot scaling and reducing customer traffic errors. Additionally, the Vertex service dependency received an update to fix the configuration, preventing recurrence of the Incident.
Customer Impact:
Customers sending traffic to gemini models in multi-region:asia experienced an increase in 5xx errors up to 100%.
Affected products: Vertex Gemini API
Affected locations: Taiwan (asia-east1), Hong Kong (asia-east2), Tokyo (asia-northeast1), Seoul (asia-northeast3), Mumbai (asia-south1), Singapore (asia-southeast1)
Incident began at 2024-12-19 08:25 and ended at 2024-12-19 11:05 (all times are US/Pacific).
We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support .
(All Times US/Pacific)
Incident Start: 19 December, 2024 08:15
Incident End: 19 December, 2024 10:55
Duration: 2 hours, 40 minutes
Affected Services and Features:
Vertex Gemini API
Regions/Zones: multi-region:asia
Description:
Google Vertex Gemini API customers sending traffic to gemini models in multi-region:asia experienced an increase in 5xx errors up to 100% for a duration of 2 hours, 40 minutes. From preliminary analysis, the root cause of the incident is stemmed from a Vertex service dependency ending up in a bad state due to a configuration process.
The situation was mitigated by increasing resources, enabling successful autopilot scaling and reducing customer traffic errors. Additionally, the Vertex service dependency received an update to fix the configuration, preventing recurrence of the Incident.
Customer Impact:
Customers sending traffic to gemini models in multi-region:asia experienced an increase in 5xx errors up to 100%.
Affected products: Vertex Gemini API
Affected locations: Taiwan (asia-east1), Hong Kong (asia-east2), Tokyo (asia-northeast1), Seoul (asia-northeast3), Mumbai (asia-south1), Singapore (asia-southeast1)
Incident began at 2024-12-19 09:30 and ended at 2024-12-19 21:30 (times are in Coordinated Universal Time (UTC)).
The issue which was isolated to AdMob traffic only has been fully resolved as of Dec 19, 2024, at 9:30 PM UTC. Ad Requests are now processing normally.
Affected products: AdMob
Incident began at 2024-12-19 09:00 (all times are US/Pacific).
Released the December 2024 spam update, which applies globally and to all languages. The rollout may take up to 1 week to complete.
Affected products: Ranking
Incident began at 2024-12-12 07:30 and ended at 2024-12-18 11:00 (all times are US/Pacific).
The rollout was complete as of December 18, 2024.
Affected products: Ranking
Incident began at 2024-12-18 04:10 and ended at 2024-12-18 08:20 (all times are US/Pacific).
This issue is now resolved
Affected products: App Distribution
Incident began at 2024-11-21 00:00 (all times are US/Pacific).
A fix for the issue is rolling out now, and should be live in all regions by the end of the week.
Affected products: App Hosting
Incident began at 2024-11-21 00:00 (all times are US/Pacific).
The App Hosting console is reporting that custom domains are in a pending state, even though they’re not. This is caused by a communication failure inside our systems that we’re working to fix. The information in the console will be correct as soon as we can correct it. The domains are working fine; this should have no impact on end users.
Affected products: App Hosting