Day: November 25, 2024

OpenAI – Elevated Error Rate for ChatGPT and API

by

Nov 25, 16:15 PST
Resolved – This issue has now been resolved.
Starting at 10:20am PT, customers experienced elevated errors on ChatGPT and API.
ChatGPT was mostly recovered by 11:55am PT, with some free plan customers continuing to experience issues until 1:20pm PT.
API performance was recovered for most customers by 1:30pm PT, with a smaller number of customers continuing to experience issues until 3:45pm PT.

Nov 25, 15:03 PST
Monitoring – We have implemented a fix for all API models with the exception of ‘gpt-4-1106-preview’, which we are continuing to work on. We are continuing to monitor performance for across all APIs as well as ChatGPT, and will post an update as soon as able.

Nov 25, 14:27 PST
Update – We have resolved issues surrounding ChatGPT. Some GPT-4 class models (excluding 4o class) accessed by API may continue to experience elevated errors. We are continuing to work towards resolution, and will provide an update as soon as able.

Nov 25, 13:21 PST
Update – We are continuing to work towards resolving the issue, and will provide an update as soon as possible.

Nov 25, 12:18 PST
Update – We are continuing to work towards resolving the issue, and will provide an update as soon as possible.

Nov 25, 11:33 PST
Update – We have identified that this issue may also cause elevated errors in the API. We are continuing to work towards implementing a fix.

Nov 25, 11:11 PST
Identified – We have identified the root cause of this issue, and are currently working to implement a fix.

Nov 25, 11:03 PST
Investigating – We are currently experiencing elevated error rates for ChatGPT. We are currently investigating.

DigitalOcean – Block Storage in SFO3

by

Nov 25, 22:05 UTC
Resolved – Our Engineering team is investigating an issue related to our ongoing SFO3 maintenance here: https://status.digitalocean.com/incidents/4kj7krrpyg3k

From 20:23 – 20:25 UTC, some services were impacted by a drop in networking. During that time, some Managed Kubernetes clusters experienced errors from the Kubernetes API and/or an increase in 5xx errors. Communication between other services and Block Storage Volumes may have been impacted as well.

The impact has been mitigated and services should be working normally at this time.

If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.

HubSpot – CRM functionality is partially degraded

by

Nov 25, 12:54 EST
Resolved – We’ve identified and addressed the root cause of the performance degradation. The incident has been fully resolved and should be working properly. No data was lost.

Nov 25, 12:28 EST
Monitoring – We’ve addressed the issue that caused performance degradation since 10:54 AM EST (UTC -05:00) on Nov 25, 2024. We’re monitoring closely to ensure the tools recover properly.

Nov 25, 12:15 EST
Identified – We’ve identified the issue that’s caused performance degradation since 10:54 AM EST (UTC -05:00) on Nov 25, 2024. We’re addressing the cause of this issue and will update this page when we have more information.

Nov 25, 11:32 EST
Investigating – We are experiencing a system-wide performance degradation that is primarily impacting search, indexing, exports, and data synchronization of CRM records, resulting in a 1% error rate.

While most of the application remains functional, users may encounter increased error rates and slower loading times, particularly in areas that interact with CRM records. These issues can typically be resolved with a page refresh.

Our team is actively investigating and working to resolve the underlying cause of these performance issues.

GitHub – Disruption with GitHub Search

by

Nov 25, 15:25 UTC
Resolved – Between 13:30 and 15:00 UTC, repository searches were timing out for most users. The ongoing efforts from the similar incident last week helped uncover the main contributing factors. We have deployed short-term mitigations and identified longer term work to proactively identify and limit resource-intensive searches.

Nov 25, 15:24 UTC
Update – Search is now operating normally. We are declaring this issue resolved.

Nov 25, 15:10 UTC
Update – We are now observing signs of complete recovery in search. We will continue to monitor and assess.

Nov 25, 14:40 UTC
Update – We’re observing signs of recovery in search, we will continue to monitor.

Next update within 15 minutes.

Nov 25, 14:10 UTC
Update – We are seeing failures in repos, users, discussions, and wikis search. Customers may see failing searches and searching by topic may fail to load.

Code and issues search continue to be available.

The team is investigating, next update in 30m.

Nov 25, 13:58 UTC
Update – We are seeing failures in search. Customers may see failing searches and searching by topic may fail to load.

Nov 25, 13:57 UTC
Investigating – We are currently investigating this issue.

Notion – Database Automation Degradation in Notion

by

Nov 25, 04:24 PST
Resolved – Our team has deployed a fix, and the system is now functioning as expected.

Nov 25, 04:17 PST
Update – We are continuing to investigate this issue.

Nov 25, 04:17 PST
Investigating – Notion is experiencing a degradation in Database automations, causing delays in trigger execution.

GitHub – Disruption with some GitHub services

by

Nov 25, 12:17 UTC
Resolved – On November 25th, 2024 between 10:38 UTC and 12:00 UTC the Claude model for GitHub Copilot Chat experienced degraded performance. During the impact, all requests to Claude would result in an immediate error to the user. This was due to upstream errors with one of our infrastructure providers, which have since been mitigated.

We are working with our infrastructure providers to reduce time to detection and implement additional failover options to mitigate issues like this one in the future.

Nov 25, 12:17 UTC
Update – This incident has been mitigated; we are now seeing requests succeeding to the Claude 3.5 Sonnet model in Copilot.

Nov 25, 11:44 UTC
Update – The team is continuing to investigate errors using the Claude 3.5 Sonnet v2 model and has engaged our partners.

All requests to this model are failing, but other Copilot models are functional and can be used as an alternative.

Nov 25, 11:00 UTC
Update – Users cannot use the Claude 3.5 Sonnet model in GitHub Copilot currently, in both VS Code and GitHub.com chat. The team is investigating.

Nov 25, 10:51 UTC
Investigating – We are currently investigating this issue.