Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support .

(All Times US/Pacific)

Incident Start: 19 December, 2024 08:15

Incident End: 19 December, 2024 10:55

Duration: 2 hours, 40 minutes

Affected Services and Features:

Vertex Gemini API

Regions/Zones: multi-region:asia

Description:

Google Vertex Gemini API customers sending traffic to gemini models in multi-region:asia experienced an increase in 5xx errors up to 100% for a duration of 2 hours, 40 minutes. From preliminary analysis, the root cause of the incident is stemmed from a Vertex service dependency ending up in a bad state due to a configuration process.

The situation was mitigated by increasing resources, enabling successful autopilot scaling and reducing customer traffic errors. Additionally, the Vertex service dependency received an update to fix the configuration, preventing recurrence of the Incident.

Customer Impact:

Customers sending traffic to gemini models in multi-region:asia experienced an increase in 5xx errors up to 100%.