Google Ads – RESOLVED: We’re aware of a problem with AdMob affecting a majority of users

Incident began at 2024-12-19 09:30 and ended at 2024-12-19 21:30 (times are in Coordinated Universal Time (UTC)).

The problem with AdMob has been resolved. We apologize for the inconvenience and thank you for your patience and continued support.

The issue which was isolated to AdMob traffic only has been fully resolved as of Dec 19, 2024, at 9:30 PM UTC. Ad Requests are now processing normally.


Affected products: AdMob

OpenAI – Invalid Structured Output Responses

Dec 19, 13:56 PST
Resolved – Between 12:26 PM and 01:52 PM PST users were experiencing invalid responses with structured outputs. We rolled out a fix and this issue is now resolved.

Dec 19, 13:25 PST
Investigating – We are currently investigating an issue with invalid responses from structured outputs and will provide more updates shortly.

OpenAI – Some users may not receive verification emails for login or signup with username and password.

Dec 19, 12:30 PST
Resolved – Between 7:06 AM and 12:12pm PST, users were unable to signup using their username and password. This was an outage caused by a third-party system and the issue is now fully resolved.

Dec 19, 11:55 PST
Update – We are currently looking into this issue. As a result, new user signups with a username and password are temporarily unavailable. We will share additional updates as soon as possible.

Dec 19, 10:07 PST
Update – We are continuing to investigate this issue. This issue is preventing new users from signing up with a username and password. As a workaround, users can still signup using their Google, Microsoft or Apple Account.

Dec 19, 09:47 PST
Update – We are continuing to investigate this issue. This issue is preventing new users from signing up with a username and password.

Dec 19, 08:01 PST
Update – This issue also impacts users signing up using a username and password.

Dec 19, 07:43 PST
Update – We are continuing to investigate this issue.

Dec 19, 07:24 PST
Investigating – We’re aware that some users are not receiving verification emails when logging into ChatGPT, leaving them blocked. We’re currently investigating.

Snowflake – AWS – US East (N. Virginia): INC0122875

Dec 19, 07:53 PST
ResolvedCurrent status: We’ve implemented the fix for this issue and monitored the environment to confirm that service was restored. If you experience additional issues or have questions, please open a support case via Snowflake Community.

Customer experience: Customers hosted in the specified regions may have experienced delays while attempting to execute queries. Affected customers might have encountered query failures.

Incident start time: 12:10 UTC December 19, 2024
Incident end time: 13:40 UTC December 19, 2024

Preliminary root cause: A limited number of metadata database storage servers experienced connectivity issues in the region.

A root cause analysis (RCA) document will be published within seven business days.

Dec 19, 06:38 PST
MonitoringCurrent status: We’ve implemented the fix for this issue, and we’ll continue to monitor the environment until we’re confident all services are functioning properly.

Customer experience: Customers hosted in the specified regions may have experienced delays while attempting to execute queries. Affected customers might have encountered query failures.

Incident start time: 12:10 UTC December 19, 2024
Incident end time: 13:40 UTC December 19, 2024

Preliminary root cause: A limited number of metadata database storage servers experienced connectivity issues in the region.

Dec 19, 05:45 PST
InvestigatingCurrent status: We’re investigating an issue with Snowflake Data Cloud. We’ll provide an update within 60 minutes.

Customer experience: Customers hosted in the specified regions may experience delays while attempting to execute queries. Affected customers may encounter query failures.

Incident start time: 12:10 UTC December 19, 2024

Supabase – Maintenance required on Supavisor in multiple regions.

Dec 19, 15:00 UTC
Completed – The scheduled maintenance has been completed.

Dec 19, 14:00 UTC
In progress – Scheduled maintenance is currently in progress. We will provide updates as necessary.

Dec 18, 13:44 UTC
Scheduled – We will be performing maintenance on Supavisor on 2024-12-19 at 14:00UTC.

The following regions will be affected:

* ap-northeast-1
* ap-south-1
* ap-southeast-2
* eu-west-1
* eu-north-1

Connections to Supavisor in the effected regions may be temporarily disconnected during this period as we perform this required maintenance.