The __instancecheck__ magic should be defined on the metaclass, not on the class itself. See the docs:
Note that these methods are looked up on the type (metaclass) of a class. They cannot be defined as class methods in the actual class.
The curious behavior you're seeing is by misplacing the isinstance arguments. You wrote:
isinstance(a, a)
But it should really be:
isinstance(a, A)
Now, here's the comedy of errors: if you've mistakenly defined __instancecheck__ as a method on a normal class, instances of the class can accidentally be used as the second argument of isinstance (normally the second arg would be the class object). Python doesn't really care if the second argument's type is a class or a metaclass, because that's just duck-typing in action.
Note that isinstance coerces the return value of __instancecheck__ to boolean. Invoking directly with a.__instancecheck__(a) will return the None.