The documentation for git checkout is silent on how it behaves when master is ambiguous. Looking at the source code (I skimmed quickly so I might be wrong), it looks like git checkout assumes that the provided name (e.g., master) is a branch name until it discovers that there is no ref in refs/heads/* for the given name.
So the proper way to check out a branch when the revision is ambiguous is to leave off the refs/heads/, e.g., git checkout master.
Branch vs. revision
Note that there is a subtle but important difference between specifying a branch and specifying a revision (or other object). For commands and options that take a revision (or general object), specifying master is the same as specifying refs/heads/master unless master is ambiguous. It's also the same as specifying master^0, or the SHA1 that master is pointing to, etc.
For commands and options that take a branch (e.g., git branch, or the --branches option to commands like git log), specifying master is not the same as specifying refs/heads/master. In these cases, the full string refs/heads/master is interpreted as the name of the branch, causing Git to create/examine/update the ref named refs/heads/refs/heads/master instead of refs/heads/master.
The git checkout command is versatile, which is handy but can cause confusion in cases like master vs. refs/heads/master. When you specify master, and a ref named refs/heads/master exists, git checkout assumes you meant the master branch, not the revision that master is pointing to. When you specify refs/heads/master, and a ref named refs/heads/refs/heads/master does not exist, then git checkout assumes you meant the revision that master is pointing to, not a branch named refs/heads/master (thus you get a detached HEAD).
Checking out the non-branch when ambiguous
If you want to check out some other ref whose short name is also master (e.g. a tag named master), you'll have to spell out the full ref name (e.g., git checkout refs/tags/master) or spell the revision in a way that can't be interpreted as a valid branch name (e.g., git checkout master^0). The latter causes git checkout to follow the disambiguation rules described in git help revisions:
When ambiguous, a <refname> is disambiguated by taking the first match in the following rules:
- If
$GIT_DIR/<refname> exists, that is what you mean (this is usually useful only for HEAD, FETCH_HEAD, ORIG_HEAD, MERGE_HEAD and CHERRY_PICK_HEAD);
- otherwise,
refs/<refname> if it exists;
- otherwise,
refs/tags/<refname> if it exists;
- otherwise,
refs/heads/<refname> if it exists;
- otherwise,
refs/remotes/<refname> if it exists;
- otherwise,
refs/remotes/<refname>/HEAD if it exists.
Of course the result will be a detached HEAD, but that always happens when you check out a non-branch.