Google Cloud – RESOLVED: We are experiencing an issue with Chronicle Security

Incident began at 2025-01-08 09:36 and ended at 2025-01-08 11:43 (all times are US/Pacific).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 3 January 2025 12:14

Incident End: 8 January 2025 11:08

Duration: 5 days, 10 hours, 55 minutes

Affected Services and Features:

Google SecOps (Chronicle Security) – SOAR Permissions

Regions/Zones:
europe, europe-west12, europe-west2, europe-west3, europe-west6, europe-west9, asia-northeast1, asia-south1, asia-southeast1, australia-southeast1, me-central1, me-central2, me-west1, northamerica-northeast2, southamerica-east1

Description:

Google SecOps (Chronicle Security) experienced an increase in permission errors for non-admin users accessing SOAR cases. From preliminary analysis, the issue was due to a software defect introduced by a recent service update that had been rolled out to non-US regions.

The issue was fully mitigated once the affected service update was rolled back, restoring service for all affected users.

Customer Impact:

  • When a non-admin user attempted to access the SOAR cases view, they received a 403 error.


Affected products: Chronicle Security

Affected locations: Tokyo (asia-northeast1), Mumbai (asia-south1), Singapore (asia-southeast1), Sydney (australia-southeast1), Multi-region: europe, Turin (europe-west12), London (europe-west2), Frankfurt (europe-west3), Zurich (europe-west6), Paris (europe-west9), Doha (me-central1), Dammam (me-central2), Tel Aviv (me-west1), Toronto (northamerica-northeast2), São Paulo (southamerica-east1)

Google Cloud – RESOLVED: We are experiencing an issue with Chronicle Security

Incident began at 2025-01-08 09:36 and ended at 2025-01-08 11:43 (all times are US/Pacific).

Mini Incident Report

We apologize for the inconvenience this service disruption/outage may have caused. We would like to provide some information about this incident below. Please note, this information is based on our best knowledge at the time of posting and is subject to change as our investigation continues. If you have experienced impact outside of what is listed below, please reach out to Google Cloud Support using https://cloud.google.com/support.

(All Times US/Pacific)

Incident Start: 3 January 2025 12:14

Incident End: 8 January 2025 11:08

Duration: 5 days, 10 hours, 55 minutes

Affected Services and Features:

Google SecOps (Chronicle Security) – SOAR Permissions

Regions/Zones:
europe, europe-west12, europe-west2, europe-west3, europe-west6, europe-west9, asia-northeast1, asia-south1, asia-southeast1, australia-southeast1, me-central1, me-central2, me-west1, northamerica-northeast2, southamerica-east1

Description:

Google SecOps (Chronicle Security) experienced an increase in permission errors for non-admin users accessing SOAR cases. From preliminary analysis, the issue was due to a software defect introduced by a recent service update that had been rolled out to non-US regions.

The issue was fully mitigated once the affected service update was rolled back, restoring service for all affected users.

Customer Impact:

  • When a non-admin user attempted to access the SOAR cases view, they received a 403 error.


Affected products: Chronicle Security

Affected locations: Tokyo (asia-northeast1), Mumbai (asia-south1), Singapore (asia-southeast1), Sydney (australia-southeast1), Multi-region: europe, Turin (europe-west12), London (europe-west2), Frankfurt (europe-west3), Zurich (europe-west6), Paris (europe-west9), Doha (me-central1), Dammam (me-central2), Tel Aviv (me-west1), Toronto (northamerica-northeast2), São Paulo (southamerica-east1)

OpenAI – High error rates for Vision API on gpt-4o-2024-05-13 model

Jan 8, 13:05 PST
Resolved – Between 11:33 AM and 11:56 AM PST users were experiencing elevated error rates with Vision API for the gpt-4o-2025-05-13 model. We rolled out a fix and this issue is now resolved.

Jan 8, 11:59 PST
Monitoring – A fix has been implemented and we are monitoring the results

Jan 8, 11:46 PST
Investigating – We are currently experiencing degraded performance for the Vision API on gpt-4o-2024-05-13 model. We are currently investigating and will post an update as soon as possible.

Medium – Stats disruption

Jan 8, 11:02 PST
Resolved – This incident has been resolved.

Jan 8, 11:02 PST
Update – This issue has been resolved. Please go to help.medium.com if you still have issues.

Dec 20, 15:52 PST
Identified – Claps are now being restored to the story and stats pages. We continue to investigate further issues. We appreciate your patience.

Dec 20, 08:42 PST
Investigating – We are experiencing a prolonged delay in all data services. This is most evident in your stats, including views, reads, claps, audience, and earnings. No data or earnings is lost – it is just a delay. Please bear with us