Day: December 11, 2024

DigitalOcean – Cloud Control Panel Logins for a Subset of Users

by

Dec 11, 23:33 UTC
Resolved – Our Engineering team has resolved the issue impacting prohibiting a subset of users from logging in, and the login flow is now operating normally.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

Dec 11, 22:53 UTC
Monitoring – Our Engineering team has implemented a fix for the issue prohibiting a subset of users from logging in. The team is monitoring the situation, and we will share an update once this is fully resolved.

Dec 11, 19:33 UTC
Investigating – Our Engineering team is currently investigating an issue prohibiting users from logging in if they are flagged for an additional security challenge.

We are actively reviewing the root cause and working closely to resolve the issue as quickly as possible.

We apologize for the inconvenience and will provide updates as more information becomes available. Thank you for your patience.

OneLogin – OneLogin Engineers have identified an issue with the Admin Service – Bulk Operations. We are working to quickly restore service.

by
December 11, 2024 2:38AM PST
Identified – OneLogin Engineers have identified an issue with the Admin Service – Bulk Operations. We are working to quickly restore service.

December 11, 2024 4:17AM PST
Investigating – We continue to investigate an issue impacting the Admin service and bulk operations.

December 11, 2024 4:55AM PST
Investigating – We have temporarily suspended the Admin UI Bulk Operations functionality while we continue to investigate

December 11, 2024 5:43AM PST
Investigating – Suspension of bulk operations via the UI is still in place as we continue to investigate the issue

December 11, 2024 6:22AM PST
Investigating – We continue to investigate the issue with bulk operations. Next update will be in 60 minutes

December 11, 2024 7:17AM PST
Identified – The root cause has been identified and the team is currently engaged with finalizing a fix.

December 11, 2024 8:18AM PST
Identified – The root cause has been identified and the team is currently engaged with finalizing a fix. Next update will be in 60 minutes.

December 11, 2024 9:20AM PST
Identified – The root cause has been identified and the team is currently engaged with finalizing a fix. Next update will be in 60 minutes.

December 11, 2024 10:30AM PST
Identified – The root cause has been identified and the team is currently finalizing testing of the fix.

December 11, 2024 11:21AM PST
Monitoring – The fix for bulk operations has been deployed, we are currently monitoring.

December 11, 2024 12:01PM PST
Resolved – The bulk update tool incident is now resolved.

Trello – Users aren’t able to create new repeats or run existing ones when using the Card Repeater Power-Up

by

Dec 11, 12:59 EST
Resolved – A fix was released for the Card Repeater Power-up, and the repeats and new ones are working as intended.

Dec 11, 11:35 EST
Identified – Our recent change hasn’t fixed the existing issue with the Card Repeater Power-Up. We’ll keep investigating the problem, and a new update will be provided soon.

Dec 11, 11:24 EST
Monitoring – A fix has been implemented and we’re currently monitoring it.

Dec 11, 10:45 EST
Identified – We’ve identified the issue that’s causing issues with the repeats, and we’re working now on a fix.

Dec 11, 10:27 EST
Investigating – We’re currently investigating reports that users can’t create new repeats while using the Card Repeater and that existing repeats aren’t being executed.

GitLab – Duo Chat is not working for most cases in VSCode and JetBrains

by
December 10, 2024 15:55 UTC
Investigating – Users are experiencing issues accessing Duo Chat functionality within VSCode. Our engineering team is actively investigating the root cause in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18980

December 10, 2024 16:17 UTC
Investigating – No material updates at this time. Our team is still investigating the issue.

December 10, 2024 16:47 UTC
Investigating – No material updates at this time. Our team is continuing to investigate the issue.

December 10, 2024 16:52 UTC
Investigating – Current impact is identifying that Duo Chat is not working for most cases in VSCode and JetBrains. We’re still investigating and will provide updates as more information becomes available.

December 10, 2024 19:31 UTC
Investigating – Our team is still investigating the issue. We’ll provide additional updates as more information becomes available.

December 10, 2024 21:17 UTC
Identified – Our team has identified the issue and is working on a fix. Duo Chat may be temporarily unavailable in editor extensions until a fix is merged. We will provide an update once the fix has been applied.

December 10, 2024 22:26 UTC
Identified – This update is to clarify the scope of affected users. The incident only affects workflows configured using OAuth authentication. Users who authenticate using an access token remain unaffected.

December 11, 2024 02:43 UTC
Identified – No material update to report. Working toward getting identified fix applied. ETA 6-8 hours. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18980

December 11, 2024 11:05 UTC
Identified – The deployment to production has started and we are expecting it to complete in approximately 1 hour. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18980

December 11, 2024 11:51 UTC
Monitoring – The fix has been deployed to production and we are no longer seeing these errors. We will continue monitoring. More details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18980

December 11, 2024 13:37 UTC
Resolved – This incident has been resolved and Duo Chat in IDEs is now fully operational. More information can be found in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18980

GitLab – Runner registration returning 500 errors

by
December 11, 2024 02:22 UTC
Identified – We are seeing 500 errors with runner registrations. We have identified the cause, and are working on a fix.
More details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 02:50 UTC
Identified – Work continues for a fix to the identified cause.
More details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 02:59 UTC
Identified – A workaround has been identified to register a runner using an authentication token: https://docs.gitlab.com/runner/register/#register-with-a-runner-authentication-token Incident only impacting the deprecated runner registration method.

December 11, 2024 03:39 UTC
Identified – Work continues to fix the deprecated runner registration method.
More details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18984 Workaround: register runner with an authentication token.

December 11, 2024 04:10 UTC
Identified – Work to fix the deprecated runner registration method continues. Workaround: register runner with an authentication token.
More details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 04:36 UTC
Identified – As a workaround is available (using an authentication token to register a new runner) the status of this incident has been downgraded to ‘Partial Service Disruption’

December 11, 2024 04:56 UTC
Identified – A fix for this problem has been merged into the codebase – it should become available on GitLab.com in the next 6-7 hours.
For more information please see https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 11:06 UTC
Identified – The deployment to production has started and we are expecting it to complete in approximately 1 hour. More details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 11:47 UTC
Monitoring – The fix has been deployed to production and we are no longer seeing these errors. We will continue monitoring. More details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 13:34 UTC
Resolved – This incident has been resolved and runner registration is now fully operational. More information can be found in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18984