Let, $f(z) = f(x,y) = u(x,y) + iv(x,y)$ be a complex function.

Now, In this question, $u(x,y)= x^2 + ay^2$ and $v(x,y)= bxy$

So, $\frac{\partial u}{\partial x}=2x, \; \frac{\partial^2 u}{\partial x^2}=2, \; \frac{\partial u}{\partial y}=2ay, \;\frac{\partial^2 u}{\partial y^2} =2a\;$ and

$\frac{\partial v}{\partial x}=by, \; \frac{\partial^2 v}{\partial x^2}=0, \; \frac{\partial u}{\partial y}=bx, \;\frac{\partial^2 u}{\partial y^2} =0\;$

Now, If $f(z)$ is analytic then $\frac{\partial^2 u}{\partial x^2}$ $+$ $\frac{\partial^2 u}{\partial y^2} = 0$ and $\frac{\partial u}{\partial x}= \frac{\partial v}{\partial y}$

So, $2+2a=0$ and $2x=bx$

$\Rightarrow$ $a=-1, \; b= 2$

So, Answer should be $(B)$

Reference :- https://www3.nd.edu/~atassi/Teaching/ame60612/Notes/analytic_functions.pdf