Status Tracker v0.23
Nov 26, 12:00 UTC
Resolved – This incident has been resolved.
Nov 26, 11:39 UTC
Monitoring – A fix has been implemented and we are monitoring the backlog of delayed certificates, which is decreasing.
Once the backlog of the affected domains is clear we will mark the incident as resolved
Nov 26, 11:24 UTC
Identified – The issue has been identified and a fix is being implemented.
Nov 26, 11:12 UTC
Investigating – We are currently investigating an issue causing new SSL certificates to be delayed.
Nov 25, 16:15 PST
Resolved – This issue has now been resolved.
Starting at 10:20am PT, customers experienced elevated errors on ChatGPT and API.
ChatGPT was mostly recovered by 11:55am PT, with some free plan customers continuing to experience issues until 1:20pm PT.
API performance was recovered for most customers by 1:30pm PT, with a smaller number of customers continuing to experience issues until 3:45pm PT.
Nov 25, 15:03 PST
Monitoring – We have implemented a fix for all API models with the exception of ‘gpt-4-1106-preview’, which we are continuing to work on. We are continuing to monitor performance for across all APIs as well as ChatGPT, and will post an update as soon as able.
Nov 25, 14:27 PST
Update – We have resolved issues surrounding ChatGPT. Some GPT-4 class models (excluding 4o class) accessed by API may continue to experience elevated errors. We are continuing to work towards resolution, and will provide an update as soon as able.
Nov 25, 13:21 PST
Update – We are continuing to work towards resolving the issue, and will provide an update as soon as possible.
Nov 25, 12:18 PST
Update – We are continuing to work towards resolving the issue, and will provide an update as soon as possible.
Nov 25, 11:33 PST
Update – We have identified that this issue may also cause elevated errors in the API. We are continuing to work towards implementing a fix.
Nov 25, 11:11 PST
Identified – We have identified the root cause of this issue, and are currently working to implement a fix.
Nov 25, 11:03 PST
Investigating – We are currently experiencing elevated error rates for ChatGPT. We are currently investigating.
Nov 25, 22:05 UTC
Resolved – Our Engineering team is investigating an issue related to our ongoing SFO3 maintenance here: https://status.digitalocean.com/incidents/4kj7krrpyg3k
From 20:23 – 20:25 UTC, some services were impacted by a drop in networking. During that time, some Managed Kubernetes clusters experienced errors from the Kubernetes API and/or an increase in 5xx errors. Communication between other services and Block Storage Volumes may have been impacted as well.
The impact has been mitigated and services should be working normally at this time.
If you continue to experience problems, please open a ticket with our support team. We apologize for any inconvenience.
Nov 25, 12:54 EST
Resolved – We’ve identified and addressed the root cause of the performance degradation. The incident has been fully resolved and should be working properly. No data was lost.
Nov 25, 12:28 EST
Monitoring – We’ve addressed the issue that caused performance degradation since 10:54 AM EST (UTC -05:00) on Nov 25, 2024. We’re monitoring closely to ensure the tools recover properly.
Nov 25, 12:15 EST
Identified – We’ve identified the issue that’s caused performance degradation since 10:54 AM EST (UTC -05:00) on Nov 25, 2024. We’re addressing the cause of this issue and will update this page when we have more information.
Nov 25, 11:32 EST
Investigating – We are experiencing a system-wide performance degradation that is primarily impacting search, indexing, exports, and data synchronization of CRM records, resulting in a 1% error rate.
While most of the application remains functional, users may encounter increased error rates and slower loading times, particularly in areas that interact with CRM records. These issues can typically be resolved with a page refresh.
Our team is actively investigating and working to resolve the underlying cause of these performance issues.
Nov 25, 15:25 UTC
Resolved – Between 13:30 and 15:00 UTC, repository searches were timing out for most users. The ongoing efforts from the similar incident last week helped uncover the main contributing factors. We have deployed short-term mitigations and identified longer term work to proactively identify and limit resource-intensive searches.
Nov 25, 15:24 UTC
Update – Search is now operating normally. We are declaring this issue resolved.
Nov 25, 15:10 UTC
Update – We are now observing signs of complete recovery in search. We will continue to monitor and assess.
Nov 25, 14:40 UTC
Update – We’re observing signs of recovery in search, we will continue to monitor.
Next update within 15 minutes.
Nov 25, 14:10 UTC
Update – We are seeing failures in repos, users, discussions, and wikis search. Customers may see failing searches and searching by topic may fail to load.
Code and issues search continue to be available.
The team is investigating, next update in 30m.
Nov 25, 13:58 UTC
Update – We are seeing failures in search. Customers may see failing searches and searching by topic may fail to load.
Nov 25, 13:57 UTC
Investigating – We are currently investigating this issue.
Nov 25, 04:24 PST
Resolved – Our team has deployed a fix, and the system is now functioning as expected.
Nov 25, 04:17 PST
Update – We are continuing to investigate this issue.
Nov 25, 04:17 PST
Investigating – Notion is experiencing a degradation in Database automations, causing delays in trigger execution.
Nov 25, 12:17 UTC
Resolved – On November 25th, 2024 between 10:38 UTC and 12:00 UTC the Claude model for GitHub Copilot Chat experienced degraded performance. During the impact, all requests to Claude would result in an immediate error to the user. This was due to upstream errors with one of our infrastructure providers, which have since been mitigated.
We are working with our infrastructure providers to reduce time to detection and implement additional failover options to mitigate issues like this one in the future.
Nov 25, 12:17 UTC
Update – This incident has been mitigated; we are now seeing requests succeeding to the Claude 3.5 Sonnet model in Copilot.
Nov 25, 11:44 UTC
Update – The team is continuing to investigate errors using the Claude 3.5 Sonnet v2 model and has engaged our partners.
All requests to this model are failing, but other Copilot models are functional and can be used as an alternative.
Nov 25, 11:00 UTC
Update – Users cannot use the Claude 3.5 Sonnet model in GitHub Copilot currently, in both VS Code and GitHub.com chat. The team is investigating.
Nov 25, 10:51 UTC
Investigating – We are currently investigating this issue.
Nov 24, 22:04 PST
Completed – The scheduled maintenance has been completed.
Nov 24, 21:38 PST
Verifying – Verification is currently underway for the maintenance items.
Nov 24, 21:31 PST
In progress – Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 29, 21:04 PDT
Scheduled – Statuspage has migrated all customer traffic to a more secure infrastructure. A very small number of customers (less than 0.1% of the total customer base) are currently blocked due to unsupported custom configurations. We will reach out to these customers with the necessary action items. If you receive an email from us but you are unsure about which domain is affected, please get in touch with our support team.
After 24th Nov 2024, the older infrastructure will be retired. There is no downtime risk for customers who have already migrated. Customers who have received notifications regarding the required updates will not encounter issues as long as the necessary actions are taken. Otherwise, they may face problems related to TLS termination or a complete site outage.
Should you experience any issues, please contact our support team through support.atlassian.com.
Nov 24, 13:35 EST
Resolved – Our systems are functional and we are closing out this incident.
Nov 24, 11:11 EST
Investigating – We are investigating reports of issues with our systems.
Nov 23, 23:23 PST
Resolved – This incident has been resolved.
Nov 23, 22:01 PST
Identified – Some users on the Android Notion mobile app may not be able to use the templates. The issue has been identified and a fix is being deployed.