Replace sin2x with 1 - cos2x, then we get the inequality
cos2x + (1-a)cosx < a2, Let f(x) = cos2x + (1-a)cosx and examine the extreme values.
f'(x) = -2cosxsinx + (a-1)sinx = 0 --> sinx = 0 or cosx = (a-1)/2
sinx = 0 means cosx = -1 or 1(also note that these are the boundary values of cosx, so we need not to check them,too; otherwise we would have to check them). Let's try these:
Replace cosx = (a-1)/2 --> f(x) = -(a-1)2/4 and f(x) < a2 clearly for all x
Replace cosx = 1 --> f(x) = 1 + (1-a) --> 2-a < a2 and it is true for all a < -2
Replace cosx = -1 --> f(x) = 1 + (a-1) < a2, and it is true for all integer a not equal 1 or 0
Then we have an upper bound a < -2, and the greatest negative integer that a can take is -3