Github

Nov 25, 15:25 UTC
Resolved – Between 13:30 and 15:00 UTC, repository searches were timing out for most users. The ongoing efforts from the similar incident last week helped uncover the main contributing factors. We have deployed short-term mitigations and identified longer term work to proactively identify and limit resource-intensive searches.

Nov 25, 15:24 UTC
Update – Search is now operating normally. We are declaring this issue resolved.

Nov 25, 15:10 UTC
Update – We are now observing signs of complete recovery in search. We will continue to monitor and assess.

Nov 25, 14:40 UTC
Update – We’re observing signs of recovery in search, we will continue to monitor.

Next update within 15 minutes.

Nov 25, 14:10 UTC
Update – We are seeing failures in repos, users, discussions, and wikis search. Customers may see failing searches and searching by topic may fail to load.

Code and issues search continue to be available.

The team is investigating, next update in 30m.

Nov 25, 13:58 UTC
Update – We are seeing failures in search. Customers may see failing searches and searching by topic may fail to load.

Nov 25, 13:57 UTC
Investigating – We are currently investigating this issue.

Nov 25, 12:17 UTC
Resolved – On November 25th, 2024 between 10:38 UTC and 12:00 UTC the Claude model for GitHub Copilot Chat experienced degraded performance. During the impact, all requests to Claude would result in an immediate error to the user. This was due to upstream errors with one of our infrastructure providers, which have since been mitigated.

We are working with our infrastructure providers to reduce time to detection and implement additional failover options to mitigate issues like this one in the future.

Nov 25, 12:17 UTC
Update – This incident has been mitigated; we are now seeing requests succeeding to the Claude 3.5 Sonnet model in Copilot.

Nov 25, 11:44 UTC
Update – The team is continuing to investigate errors using the Claude 3.5 Sonnet v2 model and has engaged our partners.

All requests to this model are failing, but other Copilot models are functional and can be used as an alternative.

Nov 25, 11:00 UTC
Update – Users cannot use the Claude 3.5 Sonnet model in GitHub Copilot currently, in both VS Code and GitHub.com chat. The team is investigating.

Nov 25, 10:51 UTC
Investigating – We are currently investigating this issue.

Nov 21, 16:48 UTC
Resolved – On November 21, 2024, between 14:30 UTC and 15:53 UTC search services at GitHub were degraded and CPU load on some nodes hit 100%. On average, the error rate was 22 requests/second and peaked at 83 requests/second. During this incident Enterprise Profile pages were slow to load and searches may have returned low quality results.

The CPU load was mitigated by redeploying portions of our web infrastructure.

We are still working to identify the cause of the increase in CPU usage and are improving our observability tooling to better expose the cause of an incident like this in the future.

Nov 21, 16:04 UTC
Update – We are seeing recovery across all searches. The team continues to closely monitor our search system and is working to fully mitigate the cause of the problems.

Nov 21, 15:33 UTC
Update – Users will notice that loading an organization profile will sometimes not work. Additionally, the site-wide search is affected, too.
This issue does not affect code or issues and pull requests searches.

Nov 21, 15:30 UTC
Investigating – We are currently investigating this issue.