Status Tracker v0.23
Incident began at 2024-12-19 11:36 and ended at 2024-12-19 17:35 (all times are US/Pacific).
The issue with Vertex Gemini API has been resolved for all affected users as of Thursday, 2024-12-19 17:30 US/Pacific.
We thank you for your patience while we worked on resolving the issue.
Affected products: Vertex Gemini API
Affected locations: Iowa (us-central1)
Incident began at 2024-12-19 08:25 and ended at 2024-12-19 11:05 (all times are US/Pacific).
We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support .
(All Times US/Pacific)
Incident Start: 19 December, 2024 08:15
Incident End: 19 December, 2024 10:55
Duration: 2 hours, 40 minutes
Affected Services and Features:
Vertex Gemini API
Regions/Zones: multi-region:asia
Description:
Google Vertex Gemini API customers sending traffic to gemini models in multi-region:asia experienced an increase in 5xx errors up to 100% for a duration of 2 hours, 40 minutes. From preliminary analysis, the root cause of the incident is stemmed from a Vertex service dependency ending up in a bad state due to a configuration process.
The situation was mitigated by increasing resources, enabling successful autopilot scaling and reducing customer traffic errors. Additionally, the Vertex service dependency received an update to fix the configuration, preventing recurrence of the Incident.
Customer Impact:
Customers sending traffic to gemini models in multi-region:asia experienced an increase in 5xx errors up to 100%.
Affected products: Vertex Gemini API
Affected locations: Taiwan (asia-east1), Hong Kong (asia-east2), Tokyo (asia-northeast1), Seoul (asia-northeast3), Mumbai (asia-south1), Singapore (asia-southeast1)
Incident began at 2024-12-19 08:25 and ended at 2024-12-19 11:05 (all times are US/Pacific).
We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support .
(All Times US/Pacific)
Incident Start: 19 December, 2024 08:15
Incident End: 19 December, 2024 10:55
Duration: 2 hours, 40 minutes
Affected Services and Features:
Vertex Gemini API
Regions/Zones: multi-region:asia
Description:
Google Vertex Gemini API customers sending traffic to gemini models in multi-region:asia experienced an increase in 5xx errors up to 100% for a duration of 2 hours, 40 minutes. From preliminary analysis, the root cause of the incident is stemmed from a Vertex service dependency ending up in a bad state due to a configuration process.
The situation was mitigated by increasing resources, enabling successful autopilot scaling and reducing customer traffic errors. Additionally, the Vertex service dependency received an update to fix the configuration, preventing recurrence of the Incident.
Customer Impact:
Customers sending traffic to gemini models in multi-region:asia experienced an increase in 5xx errors up to 100%.
Affected products: Vertex Gemini API
Affected locations: Taiwan (asia-east1), Hong Kong (asia-east2), Tokyo (asia-northeast1), Seoul (asia-northeast3), Mumbai (asia-south1), Singapore (asia-southeast1)
Dec 19, 23:27 UTC
Resolved – Gitlab has mitigated the issue and we are seeing recovery and no errors during ingestion.
Dec 19, 22:41 UTC
Update – We’re seeing issues with the Gitlab integration, given Gitlab’s ongoing issues: https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/67645861f92e4e58064f7d32
Dec 19, 22:41 UTC
Investigating – We’re seeing issues with the Gitlab integration, given Gitlab’s ongoing issues: https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/67645861f92e4e58064f7d32
Incident began at 2024-12-19 09:30 and ended at 2024-12-19 21:30 (times are in Coordinated Universal Time (UTC)).
The issue which was isolated to AdMob traffic only has been fully resolved as of Dec 19, 2024, at 9:30 PM UTC. Ad Requests are now processing normally.
Affected products: AdMob
Dec 19, 13:56 PST
Resolved – Between 12:26 PM and 01:52 PM PST users were experiencing invalid responses with structured outputs. We rolled out a fix and this issue is now resolved.
Dec 19, 13:25 PST
Investigating – We are currently investigating an issue with invalid responses from structured outputs and will provide more updates shortly.
Dec 19, 12:30 PST
Resolved – Between 7:06 AM and 12:12pm PST, users were unable to signup using their username and password. This was an outage caused by a third-party system and the issue is now fully resolved.
Dec 19, 11:55 PST
Update – We are currently looking into this issue. As a result, new user signups with a username and password are temporarily unavailable. We will share additional updates as soon as possible.
Dec 19, 10:07 PST
Update – We are continuing to investigate this issue. This issue is preventing new users from signing up with a username and password. As a workaround, users can still signup using their Google, Microsoft or Apple Account.
Dec 19, 09:47 PST
Update – We are continuing to investigate this issue. This issue is preventing new users from signing up with a username and password.
Dec 19, 08:01 PST
Update – This issue also impacts users signing up using a username and password.
Dec 19, 07:43 PST
Update – We are continuing to investigate this issue.
Dec 19, 07:24 PST
Investigating – We’re aware that some users are not receiving verification emails when logging into ChatGPT, leaving them blocked. We’re currently investigating.
Incident began at 2024-12-19 09:00 (all times are US/Pacific).
Released the December 2024 spam update, which applies globally and to all languages. The rollout may take up to 1 week to complete.
Affected products: Ranking
Dec 13, 21:46 UTC
Scheduled – We will be performing scheduled maintenance in LAX (Los Angeles) datacenter on 2024-12-19 between 17:00 and 22:00 UTC.
Traffic might be re-routed from this location, hence there is a possibility of a slight increase in latency during this maintenance window for end-users in the affected region. For PNI / CNI customers connecting with us in this location, please make sure you are expecting this traffic to fail over elsewhere during this maintenance window as network interfaces in this datacentre may become temporarily unavailable.
You can now subscribe to these notifications via Cloudflare dashboard and receive these updates directly via email, PagerDuty and webhooks (based on your plan): https://developers.cloudflare.com/notifications/notification-available/#cloudflare-status.
THIS IS A SCHEDULED EVENT Dec 19, 17:00 – 22:00 UTC
Dec 18, 21:46 UTC
Scheduled – We will be performing scheduled maintenance in HNL (Honolulu) datacenter between 2024-12-19 16:00 and 2024-12-20 01:00 UTC.
Traffic might be re-routed from this location, hence there is a possibility of a slight increase in latency during this maintenance window for end-users in the affected region. For PNI / CNI customers connecting with us in this location, please make sure you are expecting this traffic to fail over elsewhere during this maintenance window as network interfaces in this datacentre may become temporarily unavailable.
You can now subscribe to these notifications via Cloudflare dashboard and receive these updates directly via email, PagerDuty and webhooks (based on your plan): https://developers.cloudflare.com/notifications/notification-available/#cloudflare-status.
THIS IS A SCHEDULED EVENT Dec 19, 16:00 UTC – Dec 20, 01:00 UTC