Month: November 2024

GitHub – Disruption with GitHub Search

by

Nov 25, 15:25 UTC
Resolved – Between 13:30 and 15:00 UTC, repository searches were timing out for most users. The ongoing efforts from the similar incident last week helped uncover the main contributing factors. We have deployed short-term mitigations and identified longer term work to proactively identify and limit resource-intensive searches.

Nov 25, 15:24 UTC
Update – Search is now operating normally. We are declaring this issue resolved.

Nov 25, 15:10 UTC
Update – We are now observing signs of complete recovery in search. We will continue to monitor and assess.

Nov 25, 14:40 UTC
Update – We’re observing signs of recovery in search, we will continue to monitor.

Next update within 15 minutes.

Nov 25, 14:10 UTC
Update – We are seeing failures in repos, users, discussions, and wikis search. Customers may see failing searches and searching by topic may fail to load.

Code and issues search continue to be available.

The team is investigating, next update in 30m.

Nov 25, 13:58 UTC
Update – We are seeing failures in search. Customers may see failing searches and searching by topic may fail to load.

Nov 25, 13:57 UTC
Investigating – We are currently investigating this issue.

Notion – Database Automation Degradation in Notion

by

Nov 25, 04:24 PST
Resolved – Our team has deployed a fix, and the system is now functioning as expected.

Nov 25, 04:17 PST
Update – We are continuing to investigate this issue.

Nov 25, 04:17 PST
Investigating – Notion is experiencing a degradation in Database automations, causing delays in trigger execution.

GitHub – Disruption with some GitHub services

by

Nov 25, 12:17 UTC
Resolved – On November 25th, 2024 between 10:38 UTC and 12:00 UTC the Claude model for GitHub Copilot Chat experienced degraded performance. During the impact, all requests to Claude would result in an immediate error to the user. This was due to upstream errors with one of our infrastructure providers, which have since been mitigated.

We are working with our infrastructure providers to reduce time to detection and implement additional failover options to mitigate issues like this one in the future.

Nov 25, 12:17 UTC
Update – This incident has been mitigated; we are now seeing requests succeeding to the Claude 3.5 Sonnet model in Copilot.

Nov 25, 11:44 UTC
Update – The team is continuing to investigate errors using the Claude 3.5 Sonnet v2 model and has engaged our partners.

All requests to this model are failing, but other Copilot models are functional and can be used as an alternative.

Nov 25, 11:00 UTC
Update – Users cannot use the Claude 3.5 Sonnet model in GitHub Copilot currently, in both VS Code and GitHub.com chat. The team is investigating.

Nov 25, 10:51 UTC
Investigating – We are currently investigating this issue.

Statuspage – Upcoming network-related changes

by

Nov 24, 22:04 PST
Completed – The scheduled maintenance has been completed.

Nov 24, 21:38 PST
Verifying – Verification is currently underway for the maintenance items.

Nov 24, 21:31 PST
In progress – Scheduled maintenance is currently in progress. We will provide updates as necessary.

May 29, 21:04 PDT
Scheduled – Statuspage has migrated all customer traffic to a more secure infrastructure. A very small number of customers (less than 0.1% of the total customer base) are currently blocked due to unsupported custom configurations. We will reach out to these customers with the necessary action items. If you receive an email from us but you are unsure about which domain is affected, please get in touch with our support team.

After 24th Nov 2024, the older infrastructure will be retired. There is no downtime risk for customers who have already migrated. Customers who have received notifications regarding the required updates will not encounter issues as long as the necessary actions are taken. Otherwise, they may face problems related to TLS termination or a complete site outage.

Should you experience any issues, please contact our support team through support.atlassian.com.

Bubble – Issues with Main Bubble Cluster

by

Nov 24, 13:35 EST
Resolved – Our systems are functional and we are closing out this incident.

Nov 24, 11:11 EST
Investigating – We are investigating reports of issues with our systems.

Snowflake – Snowsight Service Update

by

Nov 23, 20:00 PST
Completed – The scheduled maintenance has been completed.

Nov 23, 19:00 PST
In progress – Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 19, 08:48 PST
Scheduled – As part of our efforts to improve Snowflake’s cloud data platform, Snowsight will be undergoing planned maintenance, on the AWS – GovCloud (US-West) and AWS GovCloud – DoD regions, to update its underlying database to maintain system currency.

How will this affect my service?
To complete this maintenance, the Snowsight and Open Catalog UIs will briefly be unavailable for up to 15 minutes within the scheduled 60-minute window. While we’re unable to specify the exact timing of this, all other Snowflake services and features will remain fully operational during the window.

Dropbox – Scheduled Maintenance: Dropbox Paper

by

Nov 23, 15:00 UTC
Completed – The scheduled maintenance has been completed.

Nov 23, 14:00 UTC
In progress – Scheduled maintenance is currently in progress. We will provide updates as necessary.

Nov 22, 10:14 UTC
Scheduled – Paper will be performing routine maintenance on November 23, 2024 starting at 06:00 AM (Pacific). You may not be able to access Paper for a period of time, but we’ll have things back up and running as soon as possible to minimize disruption

Okta – Resolved Service Degradation

by

Okta Engineering implemented mitigating actions and restored the connectivity impacting some users in OK6. Okta Engineering will continue monitoring. Additional root cause information will be available within 5 Business days.

Notion – Some users may be unable to purchase or duplicate templates on Marketplace.

by

Nov 22, 11:05 PST
Resolved – The incident is now resolved, and Marketplace templates are functioning normally.

Nov 21, 22:43 PST
Identified – We have identified the root cause and our engineering team is actively working on a resolution. While we have recovered most templates, some remain inaccessible. We will continue to provide updates on the situation

Nov 21, 18:44 PST
Investigating – Some users may be unable to purchase or duplicate templates on Marketplace. Also, some templates may not be appearing on Marketplace. Our engineers are investigating this currently.