Day: December 19, 2024

Cloudflare – MIA (Miami) on 2024-12-20

by

Dec 19, 16:31 UTC
Scheduled – We will be performing scheduled maintenance in MIA (Miami) datacenter on 2024-12-20 between 07:00 and 13:00 UTC.

Traffic might be re-routed from this location, hence there is a possibility of a slight increase in latency during this maintenance window for end-users in the affected region. For PNI / CNI customers connecting with us in this location, please make sure you are expecting this traffic to fail over elsewhere during this maintenance window as network interfaces in this datacentre may become temporarily unavailable.

You can now subscribe to these notifications via Cloudflare dashboard and receive these updates directly via email, PagerDuty and webhooks (based on your plan): https://developers.cloudflare.com/notifications/notification-available/#cloudflare-status.

THIS IS A SCHEDULED EVENT Dec 20, 07:00 – 13:00 UTC

GitLab – Intermittent timeouts for requests from some Utah IP addresses

by
December 16, 2024 23:49 UTC
Investigating – We have received some reports of requests timing out for IP addresses originating in Salt Lake City, Utah. We are working with a third party vendor to identify the issue. Details to follow in: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

December 17, 2024 13:45 UTC
Investigating – No material updates at the moment. We continue to work with a third party vendor to get more updates and waiting for feedback from affected customers. Details to follow in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

December 17, 2024 22:37 UTC
Investigating – No material updates at this time. Requests originating from the Salt Lake City, Utah region may still see intermittent timeouts. We are working with our third party vendor to investigate further. We will provide updates as info becomes available. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

December 18, 2024 03:57 UTC
Investigating – Our Infrastructure Engineers have been working with CloudFlare and Google and have identified a routing issue between GCP “us-west3” and Cloudflare. Investigation is continuing. Please follow https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19017 for more details.

December 18, 2024 22:15 UTC
Investigating – No material updates at this time. Cloudlfare and Google are still investigating these routing issues. We will provide additional information as it becomes available. Details will be posted to: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

December 19, 2024 01:48 UTC
Monitoring – Google’s network engineers have identified the issue affecting the “us-west3” region and deployed a fix.
We will continue monitoring.
For questions and more details, please follow https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19017.

December 20, 2024 02:51 UTC
Resolved – We are now marking this incident as resolved. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

Google Cloud – RESOLVED: Vertex Gemini API and Vertex AI in Firebase customers in us-central1 are experiencing elevated errors for requests to Gemini models

by

Incident began at 2024-12-19 11:36 and ended at 2024-12-19 17:35 (all times are US/Pacific).

The issue with Vertex Gemini API has been resolved for all affected users as of Thursday, 2024-12-19 17:30 US/Pacific.

We thank you for your patience while we worked on resolving the issue.

Affected products: Vertex Gemini API

Affected locations: Iowa (us-central1)

Google Cloud – RESOLVED: We are experiencing an issue with Vertex Gemini API in Asia

by

Incident began at 2024-12-19 08:25 and ended at 2024-12-19 11:05 (all times are US/Pacific).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support .

(All Times US/Pacific)

Incident Start: 19 December, 2024 08:15

Incident End: 19 December, 2024 10:55

Duration: 2 hours, 40 minutes

Affected Services and Features:

Vertex Gemini API

Regions/Zones: multi-region:asia

Description:

Google Vertex Gemini API customers sending traffic to gemini models in multi-region:asia experienced an increase in 5xx errors up to 100% for a duration of 2 hours, 40 minutes. From preliminary analysis, the root cause of the incident is stemmed from a Vertex service dependency ending up in a bad state due to a configuration process.

The situation was mitigated by increasing resources, enabling successful autopilot scaling and reducing customer traffic errors. Additionally, the Vertex service dependency received an update to fix the configuration, preventing recurrence of the Incident.

Customer Impact:

Customers sending traffic to gemini models in multi-region:asia experienced an increase in 5xx errors up to 100%.

Affected products: Vertex Gemini API

Affected locations: Taiwan (asia-east1), Hong Kong (asia-east2), Tokyo (asia-northeast1), Seoul (asia-northeast3), Mumbai (asia-south1), Singapore (asia-southeast1)

Google Cloud – RESOLVED: We are experiencing an issue with Vertex Gemini API in Asia

by

Incident began at 2024-12-19 08:25 and ended at 2024-12-19 11:05 (all times are US/Pacific).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support .

(All Times US/Pacific)

Incident Start: 19 December, 2024 08:15

Incident End: 19 December, 2024 10:55

Duration: 2 hours, 40 minutes

Affected Services and Features:

Vertex Gemini API

Regions/Zones: multi-region:asia

Description:

Google Vertex Gemini API customers sending traffic to gemini models in multi-region:asia experienced an increase in 5xx errors up to 100% for a duration of 2 hours, 40 minutes. From preliminary analysis, the root cause of the incident is stemmed from a Vertex service dependency ending up in a bad state due to a configuration process.

The situation was mitigated by increasing resources, enabling successful autopilot scaling and reducing customer traffic errors. Additionally, the Vertex service dependency received an update to fix the configuration, preventing recurrence of the Incident.

Customer Impact:

Customers sending traffic to gemini models in multi-region:asia experienced an increase in 5xx errors up to 100%.

Affected products: Vertex Gemini API

Affected locations: Taiwan (asia-east1), Hong Kong (asia-east2), Tokyo (asia-northeast1), Seoul (asia-northeast3), Mumbai (asia-south1), Singapore (asia-southeast1)

Compass – Gitlab integration not operational

by

Dec 19, 23:27 UTC
Resolved – Gitlab has mitigated the issue and we are seeing recovery and no errors during ingestion.

Dec 19, 22:41 UTC
Update – We’re seeing issues with the Gitlab integration, given Gitlab’s ongoing issues: https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/67645861f92e4e58064f7d32

Dec 19, 22:41 UTC
Investigating – We’re seeing issues with the Gitlab integration, given Gitlab’s ongoing issues: https://status.gitlab.com/pages/incident/5b36dc6502d06804c08349f7/67645861f92e4e58064f7d32

Google Ads – RESOLVED: We’re aware of a problem with AdMob affecting a majority of users

by

Incident began at 2024-12-19 09:30 and ended at 2024-12-19 21:30 (times are in Coordinated Universal Time (UTC)).

The problem with AdMob has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

The issue which was isolated to AdMob traffic only has been fully resolved as of Dec 19, 2024, at 9:30 PM UTC. Ad Requests are now processing normally.

Affected products: AdMob

OpenAI – Invalid Structured Output Responses

by

Dec 19, 13:56 PST
Resolved – Between 12:26 PM and 01:52 PM PST users were experiencing invalid responses with structured outputs. We rolled out a fix and this issue is now resolved.

Dec 19, 13:25 PST
Investigating – We are currently investigating an issue with invalid responses from structured outputs and will provide more updates shortly.

OpenAI – Some users may not receive verification emails for login or signup with username and password.

by

Dec 19, 12:30 PST
Resolved – Between 7:06 AM and 12:12pm PST, users were unable to signup using their username and password. This was an outage caused by a third-party system and the issue is now fully resolved.

Dec 19, 11:55 PST
Update – We are currently looking into this issue. As a result, new user signups with a username and password are temporarily unavailable. We will share additional updates as soon as possible.

Dec 19, 10:07 PST
Update – We are continuing to investigate this issue. This issue is preventing new users from signing up with a username and password. As a workaround, users can still signup using their Google, Microsoft or Apple Account.

Dec 19, 09:47 PST
Update – We are continuing to investigate this issue. This issue is preventing new users from signing up with a username and password.

Dec 19, 08:01 PST
Update – This issue also impacts users signing up using a username and password.

Dec 19, 07:43 PST
Update – We are continuing to investigate this issue.

Dec 19, 07:24 PST
Investigating – We’re aware that some users are not receiving verification emails when logging into ChatGPT, leaving them blocked. We’re currently investigating.