(The following explanation turned out to be WRONG though it explained everything. See the UPDATE as follows.)
# !xxx
This works as expected because ! is in the comment.
echo # !xxx
This also works as expected because ! is also in the comment.
echo `true # !xxx`
This also works because ! is still in the comment, though it's in the `...` context.
echo `# !xxx`
Why doesn't this work?
I guess there's a little bug when Bash interprets the `...` part. In `...`, Bash always assumes (wrongly) the first WORD is a COMMAND name so it does not think ! is in a comment and so history expansion is triggered. That's to say, echo `# !xxx` is just like echo `COMMAND !xxx`.
echo `# # !xxx`
Why does this work?
As explained in #4, the first # is parsed as a COMMAND so it's just like echo `COMMAND # !xxx` so now ! is in the comment.
echo `## !xxx`
This double hash does not work either.
As explained in #4 and #5, here ## is the first WORD and it's parsed as the COMMAND name so it's also like echo `COMMAND !xxx`.
Note that, in the `...` context, the bug is only in the first round syntax parser. That's to say, even though Bash initially parses the # as a COMMAND name, it does not really run it as a command which is named #.
UPDATE 2020-03-04
The above explanation turned out to be WRONG though it explained everything. Please see the discussion in bug-bash mailing list.
I'd quote Chet's explanation here for easy reference:
> $ set -H
> $ true `# !xxx`
> bash: !xxx`: event not found
Well, the history comment character (#) is not found at the start of a
word (here # is part of the word `#), so the rest of the line is processed for history expansion.
$ true `# # !xxx`
The history comment character is found at the start of a word (here the 2nd # itself is a word) and history
expansion skips the rest of the line.
Readline history expansion knows very little about shell syntax; in
particular, it doesn't know backquotes. It never has.