Google Cloud – RESOLVED: Vertex Gemini API and Vertex AI in Firebase customers in us-central1 are experiencing elevated errors for requests to Gemini models

Incident began at 2024-12-19 11:36 and ended at 2024-12-19 17:35 (all times are US/Pacific).

The issue with Vertex Gemini API has been resolved for all affected users as of Thursday, 2024-12-19 17:30 US/Pacific.

We thank you for your patience while we worked on resolving the issue.


Affected products: Vertex Gemini API

Affected locations: Iowa (us-central1)

Compass – Gitlab integration not operational

Dec 19, 23:27 UTC
Resolved – Gitlab has mitigated the issue and we are seeing recovery and no errors during ingestion.

Dec 19, 22:41 UTC
Update – We’re seeing issues with the Gitlab integration, given Gitlab’s ongoing issues: https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/67645861f92e4e58064f7d32

Dec 19, 22:41 UTC
Investigating – We’re seeing issues with the Gitlab integration, given Gitlab’s ongoing issues: https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/67645861f92e4e58064f7d32

Google Ads – RESOLVED: We’re aware of a problem with AdMob affecting a majority of users

Incident began at 2024-12-19 09:30 and ended at 2024-12-19 21:30 (times are in Coordinated Universal Time (UTC)).

The problem with AdMob has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

The issue which was isolated to AdMob traffic only has been fully resolved as of Dec 19, 2024, at 9:30 PM UTC. Ad Requests are now processing normally.


Affected products: AdMob

OpenAI – Invalid Structured Output Responses

Dec 19, 13:56 PST
Resolved – Between 12:26 PM and 01:52 PM PST users were experiencing invalid responses with structured outputs. We rolled out a fix and this issue is now resolved.

Dec 19, 13:25 PST
Investigating – We are currently investigating an issue with invalid responses from structured outputs and will provide more updates shortly.

OpenAI – Some users may not receive verification emails for login or signup with username and password.

Dec 19, 12:30 PST
Resolved – Between 7:06 AM and 12:12pm PST, users were unable to signup using their username and password. This was an outage caused by a third-party system and the issue is now fully resolved.

Dec 19, 11:55 PST
Update – We are currently looking into this issue. As a result, new user signups with a username and password are temporarily unavailable. We will share additional updates as soon as possible.

Dec 19, 10:07 PST
Update – We are continuing to investigate this issue. This issue is preventing new users from signing up with a username and password. As a workaround, users can still signup using their Google, Microsoft or Apple Account.

Dec 19, 09:47 PST
Update – We are continuing to investigate this issue. This issue is preventing new users from signing up with a username and password.

Dec 19, 08:01 PST
Update – This issue also impacts users signing up using a username and password.

Dec 19, 07:43 PST
Update – We are continuing to investigate this issue.

Dec 19, 07:24 PST
Investigating – We’re aware that some users are not receiving verification emails when logging into ChatGPT, leaving them blocked. We’re currently investigating.