GitLab

December 19, 2024 17:31 UTC
Investigating – We are investigating reports of intermittent errors across GitLab.com and GitLab Pages. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 17:46 UTC
Identified – We believe to have identified the cause of this incident. We are working on a mitigation strategy. Users may see errors or delays when interacting with GitLab.com. Details will be posted in: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 18:09 UTC
Identified – We are still working to mitigate the issue. Users may still encounter errors or delays on GitLab.com. Details will be posted in: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 18:37 UTC
Identified – No material updates at this time. We are still working to mitigate the issue. Details will be posted in: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 19:14 UTC
Identified – We have implemented mitigating changes. Services should begin to recover. We are continuing to adjust and monitor. Details will be posted in: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 19:51 UTC
Identified – Our initial changes have helped alleviate the issue. Our efforts are still ongoing as this is not fully mitigated. Details will be posted in: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 20:31 UTC
Identified – No material updates at this time. We have seen in overall improvement but are still are still working to fully mitigate the issue. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 21:11 UTC
Identified – Our work to fully mitigate the underlying issue is still ongoing. Details will be posted in: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 21:58 UTC
Identified – Users may still see delays when interacting with GitLab.com. We are still actively working to mitigate the issue. Details will be posted in: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 19, 2024 23:09 UTC
Identified – Mitigation steps are progressing and GitLab.com responsiveness is looking good. Follow https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19033 for more details.

December 19, 2024 23:32 UTC
Monitoring – Mitigation steps have been completed. You should see delayed tasks start to recover now. Follow https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19033 for more details.

December 20, 2024 00:08 UTC
Monitoring – We are continuing to monitor the issue at this point, with further recovery being observed. Follow gitlab.com/gitlab-com/gl-infra/production/-/issues/19033 for more details.

December 20, 2024 19:47 UTC
Resolved – GitLab.com has remained stable throughout our monitoring period and we are now marking this incident as resolved. Please see the following issue for further information related to this incident: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19033

December 16, 2024 23:49 UTC
Investigating – We have received some reports of requests timing out for IP addresses originating in Salt Lake City, Utah. We are working with a third party vendor to identify the issue. Details to follow in: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

December 17, 2024 13:45 UTC
Investigating – No material updates at the moment. We continue to work with a third party vendor to get more updates and waiting for feedback from affected customers. Details to follow in: gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

December 17, 2024 22:37 UTC
Investigating – No material updates at this time. Requests originating from the Salt Lake City, Utah region may still see intermittent timeouts. We are working with our third party vendor to investigate further. We will provide updates as info becomes available. Details: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

December 18, 2024 03:57 UTC
Investigating – Our Infrastructure Engineers have been working with CloudFlare and Google and have identified a routing issue between GCP “us-west3” and Cloudflare. Investigation is continuing. Please follow https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19017 for more details.

December 18, 2024 22:15 UTC
Investigating – No material updates at this time. Cloudlfare and Google are still investigating these routing issues. We will provide additional information as it becomes available. Details will be posted to: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

December 19, 2024 01:48 UTC
Monitoring – Google’s network engineers have identified the issue affecting the “us-west3” region and deployed a fix.
We will continue monitoring.
For questions and more details, please follow https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19017.

December 20, 2024 02:51 UTC
Resolved – We are now marking this incident as resolved. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19017

December 18, 2024 15:05 UTC
Investigating – We’re experiencing saturation in our Redis cluster. The investigation is still ongoing. Users might get 500 errors intermittently. More details here: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19025

December 18, 2024 15:48 UTC
Monitoring – We have identified the cause of the issue and have taken the necessary measures to mitigate it. We are now monitoring Redis before marking the incident as resolved. More details here: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19025

December 18, 2024 16:30 UTC
Resolved – We have confirmed our mitigation efforts were successful. We are now marking this incident as resolved. Details in: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/19025

December 10, 2024 15:55 UTC
Investigating – Users are experiencing issues accessing Duo Chat functionality within VSCode. Our engineering team is actively investigating the root cause in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18980

December 10, 2024 16:17 UTC
Investigating – No material updates at this time. Our team is still investigating the issue.

December 10, 2024 16:47 UTC
Investigating – No material updates at this time. Our team is continuing to investigate the issue.

December 10, 2024 16:52 UTC
Investigating – Current impact is identifying that Duo Chat is not working for most cases in VSCode and JetBrains. We’re still investigating and will provide updates as more information becomes available.

December 10, 2024 19:31 UTC
Investigating – Our team is still investigating the issue. We’ll provide additional updates as more information becomes available.

December 10, 2024 21:17 UTC
Identified – Our team has identified the issue and is working on a fix. Duo Chat may be temporarily unavailable in editor extensions until a fix is merged. We will provide an update once the fix has been applied.

December 10, 2024 22:26 UTC
Identified – This update is to clarify the scope of affected users. The incident only affects workflows configured using OAuth authentication. Users who authenticate using an access token remain unaffected.

December 11, 2024 02:43 UTC
Identified – No material update to report. Working toward getting identified fix applied. ETA 6-8 hours. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18980

December 11, 2024 11:05 UTC
Identified – The deployment to production has started and we are expecting it to complete in approximately 1 hour. See gitlab.com/gitlab-com/gl-infra/production/-/issues/18980

December 11, 2024 11:51 UTC
Monitoring – The fix has been deployed to production and we are no longer seeing these errors. We will continue monitoring. More details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18980

December 11, 2024 13:37 UTC
Resolved – This incident has been resolved and Duo Chat in IDEs is now fully operational. More information can be found in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18980

December 11, 2024 02:22 UTC
Identified – We are seeing 500 errors with runner registrations. We have identified the cause, and are working on a fix.
More details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 02:50 UTC
Identified – Work continues for a fix to the identified cause.
More details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 02:59 UTC
Identified – A workaround has been identified to register a runner using an authentication token: https://docs.gitlab.com/runner/register/#register-with-a-runner-authentication-token Incident only impacting the deprecated runner registration method.

December 11, 2024 03:39 UTC
Identified – Work continues to fix the deprecated runner registration method.
More details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18984 Workaround: register runner with an authentication token.

December 11, 2024 04:10 UTC
Identified – Work to fix the deprecated runner registration method continues. Workaround: register runner with an authentication token.
More details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 04:36 UTC
Identified – As a workaround is available (using an authentication token to register a new runner) the status of this incident has been downgraded to ‘Partial Service Disruption’

December 11, 2024 04:56 UTC
Identified – A fix for this problem has been merged into the codebase – it should become available on GitLab.com in the next 6-7 hours.
For more information please see https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 11:06 UTC
Identified – The deployment to production has started and we are expecting it to complete in approximately 1 hour. More details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 11:47 UTC
Monitoring – The fix has been deployed to production and we are no longer seeing these errors. We will continue monitoring. More details https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 11, 2024 13:34 UTC
Resolved – This incident has been resolved and runner registration is now fully operational. More information can be found in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18984

December 5, 2024 21:18 UTC
Investigating – We see reports of some CI Jobs failing to clone repositories with an “insufficient permissions” error. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18952 for details on the investigation.

December 5, 2024 21:40 UTC
Investigating – We believe the cause for CI clone failure is related to Group-level IP restrictions. We’re reviewing internal logs to confirm. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18952 for more.

December 5, 2024 21:52 UTC
Monitoring – Configuration changes have been put in place in our infrastructure to potentially mitigate this issue. We are currently monitoring our logs and user reports for confirmation. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18952

December 5, 2024 22:32 UTC
Monitoring – We would like to clarify that this problem affected some jobs on GitLab.com groups that used the “Restrict group access by IP address” feature. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18952.

December 5, 2024 22:50 UTC
Resolved – No more reports of this error have been received. We conclude the monitoring period and consider this incident resolved. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18952 for details.

December 4, 2024 19:38 UTC
Investigating – Jobs tagged with the “gitlab-org-docker” tag are stuck in a “Pending” status as the runners are currently offline. Please see https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18945 for further details.

December 4, 2024 19:53 UTC
Investigating – The “gitlab-org-docker” tag is meant for gitlab-org projects only and not for customer workloads. As a preliminary potential fix, please remove the tag from your affected jobs and retry them. See: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18945

December 4, 2024 20:10 UTC
Investigating – We continue to investigate our Runner infrastructure to determine the cause of the issue. Please review https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18945 for full details.

December 4, 2024 20:46 UTC
Investigating – We have found traces of connectivity issues in our Runner network infrastructure. We continue our investigation. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18945 for details.

December 4, 2024 21:10 UTC
Investigating – We have potentially identified the commit that caused this disruption in the Runner network. Investigation continues. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18945 for details.

December 4, 2024 21:29 UTC
Monitoring – We have pushed a potential fix and see signs of recovery from the affected Runners. We will continue to monitor this to ensure jobs are properly picked up. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18945.

December 4, 2024 22:19 UTC
Monitoring – Runner performance metrics are back to normal levels and jobs are being properly picked up. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18945 for the full details.

December 4, 2024 22:20 UTC
Resolved – This incident is now resolved. Please make sure not to use the “gitlab-org-docker” tag for your workloads as they are intended for gitlab-org projects only. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18945.

December 4, 2024 15:16 UTC
Investigating – We are currently investigating issues with GitLab runners with Fargate driver returning “file name too long” errors. More details about this incident can be found in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18939

December 4, 2024 15:42 UTC
Identified – We’ve identified cause of the issue and are working on resolving it. More details can be found in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18939.

December 4, 2024 15:46 UTC
Monitoring – We have disabled the feature flag and are now monitoring the issue for 1 hour before marking as resolved. More details can be found in https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18939.

December 4, 2024 16:50 UTC
Resolved – As no new user reports have been received during our monitoring period we consider this incident resolved. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18939 for the full incident history.

December 3, 2024 14:04 UTC
Investigating – We are currently investigating emails being sent for older project mirrors and imports. More details about this incident can be found in gitlab.com/gitlab-com/gl-infra/production/-/issues/18929

December 3, 2024 14:47 UTC
Identified – We’ve turned off the functionality that sends out email updates temporarily for project mirrors. We are continuing to investigate this incident.

December 3, 2024 15:51 UTC
Identified – We’ve turned off the functionality relating to schedules for project mirroring. Project mirroring will be reenabled once we resolve this issue. Mirrored projects will not be updated during this time.

December 3, 2024 18:49 UTC
Monitoring – Project mirroring has been re-enabled on GitLab.com and we are monitoring to make sure no further issues arise. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18929 for further details.

December 3, 2024 20:26 UTC
Resolved – After seeing no further issues arise during our monitoring period, we are considering this incident resolved. Please review https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18929 for more details.

November 28, 2024 09:39 UTC
Investigating – The Customers Portal is currently in maintenance mode and unavailable due to a 3rd-party API outage. The billing pages in GitLab.com may be affected.

November 28, 2024 10:39 UTC
Resolved – Functionality has been restored to the Customer Portal (customers.gitlab.com). The billing pages in GitLab.com are also available.