Day: December 5, 2024

OneLogin – Release

by
December 5, 2024 10:27AM PST
Scheduled – November Release is scheduled for tonight, Dec 5th, 2024 from 7:30 pm to 11:30 pm Pacific Time. No downtime is expected for any of our services. We suggest not making any major changes until the maintenance window is completed. You can read about our release at https://onelogin.service-now.com/support?id=kb_article&sys_id=1e031d0f1bf196109a47ec29b04bcbd5

December 5, 2024 7:30PM PST
Active – Scheduled maintenance is starting.

December 5, 2024 11:30PM PST
Completed – Scheduled maintenance is complete.

Atlassian Developer – App upgrades intermittently failing causing uninstallation with inability to reinstall

by

Dec 6, 05:30 UTC
Resolved – The fix for this issue has been rolled out at 1PM 5th Dec UTC and this issue has now been resolved. If you have experienced issues updating your apps or if you notice any missing apps please attempt to reinstall the app. If you experience problems reinstalling an app, please raise a request with Atlassian support.

Dec 5, 06:10 UTC
Identified – We have identified an intermittent issue since Mon, 18 Nov 2024 14:56 GMT where attempting to update a third party app that makes use of translation packs can result in that app being uninstalled instead, with subsequent attempts to reinstall the app also failing in some cases. The error message displayed for both failed upgrades and failed uninstalls is as follows
‘Unfortunately, we’re unable to process your request. Please retry again in some time.
draw.io is known to be affected by this issue.’
We’re in the process of rolling out a fix and this should be rolled out by 6 December UTC. We recommend holding off updating your apps for now, but if you update an app and experience this issue, please attempt to reinstall the app. If you experience problems reinstalling the app, please raise a request with Atlassian support.
We apologize for the inconvenience. We will monitor the resolution and will provide an update at 11pm UTC, 5th December.

Confluence – App upgrades intermittently failing causing uninstallation with inability to reinstall

by

Dec 6, 05:30 UTC
Resolved – The fix for this issue has been rolled out at 1PM 5th Dec UTC and this issue has now been resolved. If you have experienced issues updating your apps or if you notice any missing apps please attempt to reinstall the app. If you experience problems reinstalling an app, please raise a request with Atlassian support.

Dec 5, 06:10 UTC
Identified – We have identified an intermittent issue since Mon, 18 Nov 2024 14:56 GMT where attempting to update a third party app that makes use of translation packs can result in that app being uninstalled instead, with subsequent attempts to reinstall the app also failing in some cases. The error message displayed for both failed upgrades and failed uninstalls is as follows
‘Unfortunately, we’re unable to process your request. Please retry again in some time.
draw.io is known to be affected by this issue.’
We’re in the process of rolling out a fix and this should be rolled out by 6 December UTC. We recommend holding off updating your apps for now, but if you update an app and experience this issue, please attempt to reinstall the app. If you experience problems reinstalling the app, please raise a request with Atlassian support.
We apologize for the inconvenience. We will monitor the resolution and will provide an update at 11pm UTC, 5th December.

New Relic – Delayed alert notifications for the EU region

by

Dec 6, 02:19 UTC
Resolved – Between 22:51 UTC on Wednesday, December 5th and 02:16 UTC on December 6th, some customers in the EU region may have experienced delayed or missing alert notifications. Data processing for affected data may have been impacted.

We have investigated and resolved this service interruption and impacted services have returned to normal operations.

Dec 6, 01:35 UTC
Identified – The issue has been identified and a fix is being implemented.

Some customers in the EU region may experience delayed or missing alert notifications. Data ingest is not impacted at this time.

Dec 6, 00:23 UTC
Update – We are continuing to investigate an issue where some customers in the EU region may experience delayed alert notifications. Data ingest is not impacted at this time.

Dec 5, 23:25 UTC
Update – Some customers in the EU region may intermittently experience delayed alert notifications. Data ingest is not impacted at this time.

We are continuing to investigate this issue.

Dec 5, 23:20 UTC
Investigating – Some customers in the EU region may intermittently experience delayed alert notifications. Data ingest is not impacted at this time.

DigitalOcean – Reserved IP Page in Control Panel

by

Dec 6, 00:50 UTC
Resolved – Our Engineering has confirmed full resolution of the issue. Users should now be able to manage their Reserved IPs via the Cloud Panel without issue.

If you continue to experience any issues, please open a support ticket from within your account. Thank you.

Dec 6, 00:16 UTC
Monitoring – Our Engineering team has deployed a fix for the issue with managing Reserved IPs via the Cloud Panel. Users should now be able to view the Reserved IPs page in their account.

We are currently monitoring the situation and will post an update as soon as the issue is fully resolved.

Dec 5, 23:59 UTC
Investigating – Our Engineering team is investigating an issue with managing Reserved IPs in the Cloud Panel. During this time, users may experience an error message when trying to view their Reserved IPs page at https://cloud.digitalocean.com/networking/reserved_ips. Managing Reserved IPs via the API should still be functioning.

We apologize for the inconvenience and will share an update once we have more information.

Zapier – Zap error notification emails delayed

by

Dec 5, 16:39 PST
Resolved – As of 12:30 AM UTC, the delay in notification emails has been resolved and all error notifications are being sent without a delay.

If you need further assistance, please do not hesitate to reach out to our Support Team: https://zapier.com/app/get-help.

Dec 5, 13:31 PST
Investigating – As of 7:40 am UTC, users may see a delay of up to 3 hours for error notifications regarding failed Zap Runs.

Zap Runs and tasks are proceeding as expected.

Our team is actively looking into this issue and trying to minimize its impact on your use of our service. We appreciate your patience as we work to resolve this.

If you need further assistance, please do not hesitate to reach out to our Support Team: https://zapier.com/app/get-help.

GitLab – Some CI jobs are failing to run due to insufficient permissions

by
December 5, 2024 21:18 UTC
Investigating – We see reports of some CI Jobs failing to clone repositories with an “insufficient permissions” error. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18952 for details on the investigation.

December 5, 2024 21:40 UTC
Investigating – We believe the cause for CI clone failure is related to Group-level IP restrictions. We’re reviewing internal logs to confirm. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18952 for more.

December 5, 2024 21:52 UTC
Monitoring – Configuration changes have been put in place in our infrastructure to potentially mitigate this issue. We are currently monitoring our logs and user reports for confirmation. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18952

December 5, 2024 22:32 UTC
Monitoring – We would like to clarify that this problem affected some jobs on GitLab.com groups that used the “Restrict group access by IP address” feature. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18952.

December 5, 2024 22:50 UTC
Resolved – No more reports of this error have been received. We conclude the monitoring period and consider this incident resolved. See https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18952 for details.

Netlify – Image CDN returning 502s

by

Dec 5, 20:56 UTC
Resolved – Our monitors and logging point to our Image CDN no longer responding with 502s.

Dec 5, 20:50 UTC
Monitoring – A fix has been made to address the issue causing the 502 response and are monitoring to make sure the change has resolved the issue completely.

Dec 5, 20:37 UTC
Identified – We have identified the cause of the 502s and are working towards a fix.

Dec 5, 20:23 UTC
Investigating – We are currently investigating reports of issues with our ImageCDN.