Month: November 2024

GitLab – 500 errors on user preferences pages

by
November 18, 2024 07:02 UTC
Identified – We are seeing a number of 500 errors occurring when users attempt to navigate to the user profile settings in the UI – We have identified the issue and are working on a fix. The issue is being tracked in the incident issue: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 07:18 UTC
Identified – A fix has been implemented and is being tested before being merged. For more information please see: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 07:34 UTC
Identified – A fix has been merged and it will be deployed shortly – once it has been deployed we will monitor the number of 500s occurring on the settings page to ensure the issue is resolved. For more information please see: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 07:57 UTC
Identified – The fix is awaiting deployment, which is expected to occur before 12:00 UTC. Once the fix has deployed the User Preferences page will be monitored to ensure the issue has been resolved. For more information please see: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 09:24 UTC
Identified – A fix is currently being deployed and is expected to complete in the next ~3 hours. Another update will be provided at 12:00 UTC or once the deployment has completed. For more information please see: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 11:58 UTC
Identified – The deployment process to resolve the issue is still ongoing. We will confirm once the deployment is complete or provide another progress update within ~1 hour. For more information please see: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 13:16 UTC
Identified – We have deployed on canary and the revert fixed the issue with accessing the User Preferences page. We will confirm once deployment to production is complete within the next ~40 minutes. For more information please see: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 14:07 UTC
Identified – Deployment to production is currently ongoing, but is expected to complete shortly. We will provide another update once deployment is completed. For more information please see: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 15:18 UTC
Monitoring – The fix has been deployed to production and the User Preferences page should be accessible again. We’re now monitoring to ensure the issue has been resolved. For more information please see: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

November 18, 2024 15:45 UTC
Resolved – This incident has been resolved as we are no longer seeing any issues on the User Preferences page after a period of monitoring. For more information please see: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18867

Discord – API Errors

by

Nov 16, 22:18 PST
Resolved – This incident has been resolved.

Nov 16, 22:05 PST
Monitoring – We believe we have a fix implemented and are monitoring recovery

Nov 16, 21:58 PST
Identified – We have identified the issue and are working to mitigate.

Nov 16, 21:38 PST
Update – Our team is actively investigating this issue

Nov 16, 21:03 PST
Investigating – We are currently investigating an issue affecting latency and errors on various API endpoints.

Snowflake – Snowsight Service Update

by

Nov 16, 20:00 PST
Completed – The scheduled maintenance has been completed.

Nov 16, 19:00 PST
In progress – Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 12, 08:39 PST
Scheduled – As part of our efforts to improve Snowflake’s cloud data platform, Snowsight will be undergoing planned maintenance to update its underlying database to maintain system currency.

How will this affect my service?
To complete this maintenance, the Snowsight and Open Catalog UIs will briefly be unavailable for up to 15 minutes within the scheduled 60-minute window. While we’re unable to specify the exact timing of this, all other Snowflake services and features will remain fully operational during the window.

SendGrid – SendGrid Event Delays for EU Regions

by

Nov 16, 15:54 PST
Resolved – This incident has been resolved.

Nov 16, 15:37 PST
Monitoring – A fix has been implemented and we are monitoring the results.

Nov 16, 15:06 PST
Identified – The issue has been identified and a fix is being implemented.

Nov 16, 14:34 PST
Investigating – Starting around 1:00 PM PST on Saturday, November 16th, 2024, our Engineers began investigating an issue with SendGrid Events. Customers may experience delays with Event reporting associated to mail send. The issue is causing intermittent errors and delays on Events (webhook, open and click events, Stats and Email Activity Feed events). This does not impact mail send. We are actively working to resolve the issue. We expect to provide another update in 30 minutes or as soon as more information becomes available.

Atlassian Developer – Marketplace reporting data will be delayed by 8 hours on November 16, 2024

by

Nov 16, 13:00 UTC
Completed – The scheduled maintenance has been completed.

Nov 16, 05:00 UTC
In progress – Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 15, 05:04 UTC
Update – We will be undergoing scheduled maintenance during this time.

Nov 15, 05:00 UTC
Scheduled – Marketplace reporting data will be delayed by 8 hours on November 16, 2024

Atlassian Analytics – Errors creating/editing data lake connections in Atlassian Analytics

by

Nov 15, 17:33 UTC
Resolved – We’ve deployed a fix for the issue and lake connection management in Atlassian Analytics is restored to normal operation.

Nov 15, 16:31 UTC
Identified – We’ve identified the problem and are working to deploy the fix.

Nov 15, 16:26 UTC
Investigating – We are currently investigating an issue with creating or editing data lake connections in Atlassian Analytics.

Klaviyo – Data Infrastructure Maintenance

by

Nov 14, 22:10 EST
Completed – The scheduled maintenance has been completed.

Nov 14, 20:00 EST
In progress – Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 14, 13:03 EST
Scheduled – Beginning at 8:00 PM (US Eastern Time) on Thursday, November 14, 2024, Klaviyo Engineering will perform maintenance on our data infrastructure.

During this maintenance window, Profiles deletes, updates, and merges may not be processed, and profile-related API calls may respond with errors. At the completion of the maintenance window, any failed Profiles action can be attempted again. No other Klaviyo functionality will be impacted.

Netlify – Global Edge Function Errors

by

Nov 15, 02:15 UTC
Resolved – This incident has been resolved.

Nov 15, 02:14 UTC
Update – The fix has been implemented

Nov 15, 01:42 UTC
Identified – The issue has been identified and a fix is being implemented.

Nov 15, 01:25 UTC
Investigating – We are seeing increased latency and errors with edge functiosns in Europe and the US and are currently investigating.