This error occurs if the default branch of a repository has been deleted on your GitHub Enterprise instance.

Detecting this error is simple; Git will warn you when you try to clone the repository:

git clone https://hostname/user/repo.git
# Clone a repo
# Cloning into 'repo'...
# remote: Counting objects: 66179, done.
# remote: Compressing objects: 100% (15587/15587), done.
# remote: Total 66179 (delta 46985), reused 65596 (delta 46402)
# Receiving objects: 100% (66179/66179), 51.66 MiB | 667 KiB/s, done.
# Resolving deltas: 100% (46985/46985), done.
# warning: remote HEAD refers to nonexistent ref, unable to checkout.

To fix the error, you'll need to be an administrator of the repository on your GitHub Enterprise instance. You'll want to change the default branch of the repository.

After that, you can get a list of all the available branches from the command line:

git branch -a
# Lists ALL the branches
#   remotes/origin/awesome
#   remotes/origin/more-work
#   remotes/origin/new-master

Then, you can just switch to your new branch:

git checkout new-master
# Create and checkout a tracking branch
# Branch new-master set up to track remote branch new-master from origin.
# Switched to a new branch 'new-master'