You have correctly calculated that the value of a after the i-th iteration of the inner loop is:

Where a_j0 is the value of a at the start of the j-th outer loop. The stopping condition for the inner loop is:

Which can be solved as a quadratic inequality:

Therefore the inner loop is approximately O(sqrt(a_j0 / b)). The next starting value of a satisfies:

Scaling roughly as sqrt(2b * a_j0). It would be quite tedious to compute the time complexity exactly, so let's apply the above approximations from here on:

Where a_n replaces a_j0, and t_n is the run-time of the inner loop – and of course the total time complexity is just the sum of t_n. Note that the first term is given by n = 1, and that the input value of a is defined to be a_0.
Before directly solving this recurrence, note that since the second term t_2 is already proportional to the square root of the first t_1, the latter dominates all other terms in the sum.
The total time complexity is therefore just O(sqrt(a / b)).
Update: numerical tests.
Note that, since all changes in the value of a are proportional to b, and all loop conditions are also proportional to b, the function can be "normalized" by setting b = 1 and only varying a.
Javascript test function, which measures the number of times that the inner loop executes:
function T(n)
{
let t = 0, k = 0;
while (n >= 1) {
k = 1;
while (n >= k) {
n -= k;
k++; t++;
}
}
return t;
}
Plot of sqrt(n) against T(n):

A convincing straight line which confirms that the time complexity is indeed half-power.