DigitalOcean – Droplet Event Processing in Multiple Regions

by

Dec 19, 05:03 UTC
Resolved – From 1:23 UTC to 3:04 UTC, users may have experienced issues with events being stuck or delayed, such as powering on/off, and resizing Droplets in the NYC3, AMS2, BLR1, SGP1, and SYD1 regions. Additionally, Managed Database creates were delayed in all regions.

Our Engineering team has confirmed full resolution of the issue, delayed events and new events should complete as normal now.

Thank you for your patience through this issue. If you continue to experience any issues, please open a support ticket from within your account.

Dec 19, 04:28 UTC
Update – Our Engineering team has confirmed that new events are succeeding, and we’re finalizing cleaning up any stalled events. We will post an update as soon this issue is fully resolved.

Dec 19, 03:08 UTC
Monitoring – Events in NYC3 has been enabled now and users may continue to submit new events. We are starting to see success rates with event processing go up.

We will post another update soon. Thank you for your patience.

Dec 19, 02:42 UTC
Update – Our Engineering team continues to work to mitigate the issue with event processing in multiple regions.

To help address the issue, we are temporarily disabling all Droplet-related actions in the NYC3 region for a period of 20 minutes. During this time, users in NYC3 will not be able to submit actions, such as creating, resizing, powering on/off, etc.

We apologize for the inconvenience and appreciate your patience as we work to resolve this issue. Further updates will be shared as they become available.

Dec 19, 01:45 UTC
Investigating – Our Engineering team is investigating an issue impacting Droplet event processing in multiple regions.

At this time, users may experience issues with events appearing to be stuck or delayed, such as powering on/off, and resizing Droplets in NYC3, AMS2, BLR1, SGP1, and SYD1 regions. Additionally, Managed Database creates are delayed in all regions.

We apologize for the inconvenience and will share an update once we have more information.

Jira Service Management – Customers experiencing outage in Asia Paciffic region for Assets based experiences

by

Dec 19, 02:34 UTC
Resolved – Between 12:19 UTC to 02:27 UTC we experienced degradation of assets experiences and customer and organisation profile information for some customers in the Asia Pacific region. The issue has now been resolved and the service is operating normally.

Dec 19, 02:30 UTC
Monitoring – We are monitoring traffic to confirm resolution.

Dec 19, 02:23 UTC
Identified – We have identified the fault in the system and are addressing the issue.

Dec 19, 02:22 UTC
Investigating – We notice customers getting impacted by Internal Server Errors across Asia Pacific region for Assets powered experiences.
Jira Service Management and Assets experiences are partially unavailable.

Cloudflare – LAX (Los Angeles) on 2024-12-18

by

Dec 18, 21:30 UTC
Completed – The scheduled maintenance has been completed.

Dec 18, 09:00 UTC
In progress – Scheduled maintenance is currently in progress. We will provide updates as necessary.

Dec 16, 20:55 UTC
Update – We will be performing scheduled maintenance in LAX (Los Angeles) datacenter on 2024-12-18 between 09:00 and 21:30 UTC.

Traffic might be re-routed from this location, hence there is a possibility of a slight increase in latency during this maintenance window for end-users in the affected region. For PNI / CNI customers connecting with us in this location, please make sure you are expecting this traffic to fail over elsewhere during this maintenance window as network interfaces in this datacentre may become temporarily unavailable.

You can now subscribe to these notifications via Cloudflare dashboard and receive these updates directly via email, PagerDuty and webhooks (based on your plan): https://developers.cloudflare.com/notifications/notification-available/#cloudflare-status.

Dec 13, 22:32 UTC
Scheduled – We will be performing scheduled maintenance in LAX (Los Angeles) datacenter on 2024-12-18 between 09:00 and 14:00 UTC.

Traffic might be re-routed from this location, hence there is a possibility of a slight increase in latency during this maintenance window for end-users in the affected region. For PNI / CNI customers connecting with us in this location, please make sure you are expecting this traffic to fail over elsewhere during this maintenance window as network interfaces in this datacentre may become temporarily unavailable.

You can now subscribe to these notifications via Cloudflare dashboard and receive these updates directly via email, PagerDuty and webhooks (based on your plan): https://developers.cloudflare.com/notifications/notification-available/#cloudflare-status.

GitLab – Redis Cluster Saturation

by
December 18, 2024 15:05 UTC
Investigating – We’re experiencing saturation in our Redis cluster. The investigation is still ongoing. Users might get 500 errors intermittently. More details here: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19025

December 18, 2024 15:48 UTC
Monitoring – We have identified the cause of the issue and have taken the necessary measures to mitigate it. We are now monitoring Redis before marking the incident as resolved. More details here: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19025

December 18, 2024 16:30 UTC
Resolved – We have confirmed our mitigation efforts were successful. We are now marking this incident as resolved. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19025