Let, $f(z) = f(x,y) = u(x,y) + iv(x,y)$ be a complex function.
Now, In this question, $u(x,y)= x^2 + ay^2$ and $v(x,y)= bxy$
So, $\frac{\partial u}{\partial x}=2x, \; \frac{\partial^2 u}{\partial x^2}=2, \; \frac{\partial u}{\partial y}=2ay, \;\frac{\partial^2 u}{\partial y^2} =2a\;$ and
$\frac{\partial v}{\partial x}=by, \; \frac{\partial^2 v}{\partial x^2}=0, \; \frac{\partial u}{\partial y}=bx, \;\frac{\partial^2 u}{\partial y^2} =0\;$
Now, If $f(z)$ is analytic then $\frac{\partial^2 u}{\partial x^2}$ $+$ $\frac{\partial^2 u}{\partial y^2} = 0$ and $\frac{\partial u}{\partial x}= \frac{\partial v}{\partial y}$
So, $2+2a=0$ and $2x=bx$
$\Rightarrow$ $a=-1, \; b= 2$
So, Answer should be $(B)$
Reference :- https://www3.nd.edu/~atassi/Teaching/ame60612/Notes/analytic_functions.pdf