50 ohm characteristic impedance of my coax is only "characteristic" if it's terminated with 50+0j load
That wording is incorrect. The characteristic impedance of your coax is a constant; it doesn't depend on what you attach. It depends on the shape and the material of the coax.
What does change is the impedance of the load attached to your radio.
So when the coax is connected to an antenna that is fairly reactive like (23−j300)Ω then the coax is no longer matching my radio.
The cable is still matched to your radio. Your antenna isn't matched to the cable!
Now, you're still right, we can "count" the cable towards either the radio or the antenna. If we count it towards the radio, then your antenna is "mismatched to your radio system", if we count it towards the antenna, then "the impedance of the antenna system gets transformed by the cable".
If not, how would I determine the impedance of the coax and antenna ?
In case you remember Smith charts. We use the radio impedance=cable impedance as reference impedance $Z_0$.
If we
- "include" it into the radio, then the length of the cable transforms the impedance of the radio (normalized, so that's $Z_r=1.0$) on a circle of radius $|Z_r - Z_0|=|1-1|=0$. No matter how far we walk on a circle of radius 0, we stay in the same point! So, from the antenna, we see the cable (and whatever is attached to it) as perfect 50Ω load.
- "include" it into the antenna. Antenna has impedance $\tilde Z_A= (23−j300 )Ω$, which, normalized to 50Ω reference impedance, is ca $Z_A=0.5 - j6$. We now walk on a circle of radius $|Z_A-Z_0|=\sqrt{0.5^2 + 6^2}\approx 6$, for as many half-wavelengths as your cable is long.
Result:
- In a system where we can consider the load matched to the cable, the cable length doesn't matter: from the perspective of the antenna, what is attached is always 50Ω, purely resistive.
- In a system where the load is not matched to the cable, cable length matters, and you can transform your mostly reactive impedance to any degree of purely-resistive, capacitive or reactive impedance by walking on a circle in the Smith chart. That's the whole point of that chart! So, you can even transform your heavily reactive load to a purely resistive one through the right length of 50Ω coax, and then use e.g. a resistive voltage divider or a transformer to do the matching. More often, you achieve a match by having some discrete components on either end of the cable, transforming the impedance such that it can matched through a length of different (e.g. 75Ω) coax.