Month: November 2024

DigitalOcean – NYC3 Network Maintenance

by

Nov 27, 03:00 UTC
Completed – The scheduled maintenance has been completed.

Nov 27, 01:52 UTC
Update – Our Engineering team has decided to extend the maintenance window by an hour to monitor the situation.

Maintenance is still underway and we will update once the maintenance is fully completed.

We sincerely apologize for any inconvenience this may have caused and appreciate your understanding.

Nov 27, 00:01 UTC
Update – From 22:25 – 23:15 UTC, all event-related actions including creating or deleting resources, were temporarily disabled in NYC3.

Users attempting any event-based operations across products hosted in NYC3 would have encountered error messages. While we anticipated event failures and delays, this impact period was longer than expected. We are taking steps to ensure this does not happen in future maintenances.

As of now, maintenance is still underway and we will update once the maintenance is fully completed.

We sincerely apologize for any inconvenience this may have caused and appreciate your understanding.

Nov 26, 22:00 UTC
In progress – Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 24, 22:03 UTC
Scheduled – Start: 2024-11-26 22:00 UTC
End: 2024-11-27 02:00 UTC

During the above window, our Networking team will be making changes to core networking infrastructure, to improve performance and scalability in the NYC3 region.

Expected impact:

During the maintenance window users may experience delays or failures with event processing for a brief duration on Droplets and Droplet-based services including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions related to this issue please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Canva – Users are unable to use autofill to create designs using Enterprise Integrations.

by

Nov 27, 13:59 AEDT
Resolved – This incident is now resolved. Thank you for your patience and understanding.

Nov 27, 13:25 AEDT
Identified – – Jobs created with Autofill are currently unable to complete.
– None of the other Connect API features appears to be affected.
– Canva’s Editor and Homepage are not affected by this issue.
– We have identified the cause are working to mitigate.

GitLab – Duplicate Merge Request Events

by
November 26, 2024 18:43 UTC
Identified – Users may see duplicated events on merge requests, or failed merge attempts and merge request updates.

Please see https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18904 for more information.

November 26, 2024 19:04 UTC
Monitoring – A fix has been rolled out and merge request events should no longer be duplicated.

See the production issue for more details: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18904

November 26, 2024 19:52 UTC
Resolved – We have confirmed that duplicate merge request events have stopped being created.

Please see the production issue for details: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18904

OpenAI – Elevated Errors in API

by

Nov 26, 11:30 PST
Resolved – We experienced a brief period of elevated errors for API requests between approximately 10:45 AM – 11:15 AM PT.

Anthropic – Elevated errors on Claude 3.5 Sonnet

by

Nov 26, 10:12 PST
Resolved – This incident has been resolved.

Nov 26, 08:47 PST
Monitoring – A fix has been implemented and we are monitoring the results.

Nov 26, 06:17 PST
Investigating – We are currently investigating this issue.

Supabase – Compute capacity issues observed in the eu-west-3b availability zone

by

Nov 26, 18:06 UTC
Resolved – This incident has been resolved.

Nov 23, 22:14 UTC
Identified – We have identified an issue with our cloud provider regarding insufficient compute capacity in the eu-west-3b availability zone.
We have disabled the ability to restart projects, compute or database version upgrades within eu-west-3b.

Project provisioning in eu-west-3, as well as all operations for projects located in the eu-west-3a and eu-west-3c availability zones are available.

Nov 23, 22:10 UTC
Investigating – We are currently investigating this issue.

Vercel – Delays Issuing New SSL Certificates

by

Nov 26, 12:00 UTC
Resolved – This incident has been resolved.

Nov 26, 11:39 UTC
Monitoring – A fix has been implemented and we are monitoring the backlog of delayed certificates, which is decreasing.
Once the backlog of the affected domains is clear we will mark the incident as resolved

Nov 26, 11:24 UTC
Identified – The issue has been identified and a fix is being implemented.

Nov 26, 11:12 UTC
Investigating – We are currently investigating an issue causing new SSL certificates to be delayed.

OpenAI – Elevated Error Rate for ChatGPT and API

by

Nov 25, 16:15 PST
Resolved – This issue has now been resolved.
Starting at 10:20am PT, customers experienced elevated errors on ChatGPT and API.
ChatGPT was mostly recovered by 11:55am PT, with some free plan customers continuing to experience issues until 1:20pm PT.
API performance was recovered for most customers by 1:30pm PT, with a smaller number of customers continuing to experience issues until 3:45pm PT.

Nov 25, 15:03 PST
Monitoring – We have implemented a fix for all API models with the exception of ‘gpt-4-1106-preview’, which we are continuing to work on. We are continuing to monitor performance for across all APIs as well as ChatGPT, and will post an update as soon as able.

Nov 25, 14:27 PST
Update – We have resolved issues surrounding ChatGPT. Some GPT-4 class models (excluding 4o class) accessed by API may continue to experience elevated errors. We are continuing to work towards resolution, and will provide an update as soon as able.

Nov 25, 13:21 PST
Update – We are continuing to work towards resolving the issue, and will provide an update as soon as possible.

Nov 25, 12:18 PST
Update – We are continuing to work towards resolving the issue, and will provide an update as soon as possible.

Nov 25, 11:33 PST
Update – We have identified that this issue may also cause elevated errors in the API. We are continuing to work towards implementing a fix.

Nov 25, 11:11 PST
Identified – We have identified the root cause of this issue, and are currently working to implement a fix.

Nov 25, 11:03 PST
Investigating – We are currently experiencing elevated error rates for ChatGPT. We are currently investigating.

DigitalOcean – Block Storage in SFO3

by

Nov 25, 22:05 UTC
Resolved – Our Engineering team is investigating an issue related to our ongoing SFO3 maintenance here: https://status.digitalocean.com/incidents/4kj7krrpyg3k

From 20:23 – 20:25 UTC, some services were impacted by a drop in networking. During that time, some Managed Kubernetes clusters experienced errors from the Kubernetes API and/or an increase in 5xx errors. Communication between other services and Block Storage Volumes may have been impacted as well.

The impact has been mitigated and services should be working normally at this time.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

HubSpot – CRM functionality is partially degraded

by

Nov 25, 12:54 EST
Resolved – We’ve identified and addressed the root cause of the performance degradation. The incident has been fully resolved and should be working properly. No data was lost.

Nov 25, 12:28 EST
Monitoring – We’ve addressed the issue that caused performance degradation since 10:54 AM EST (UTC -05:00) on Nov 25, 2024. We’re monitoring closely to ensure the tools recover properly.

Nov 25, 12:15 EST
Identified – We’ve identified the issue that’s caused performance degradation since 10:54 AM EST (UTC -05:00) on Nov 25, 2024. We’re addressing the cause of this issue and will update this page when we have more information.

Nov 25, 11:32 EST
Investigating – We are experiencing a system-wide performance degradation that is primarily impacting search, indexing, exports, and data synchronization of CRM records, resulting in a 1% error rate.

While most of the application remains functional, users may encounter increased error rates and slower loading times, particularly in areas that interact with CRM records. These issues can typically be resolved with a page refresh.

Our team is actively investigating and working to resolve the underlying cause of these performance issues.