Operations

Note that $a \star b$ really looks like $(a + b)^2$, but it is not actually that, so we work around that by doing this:

$a\star b = a^2 + b^2 = (a^2 \color{red} + 2ab\color{black} + b^2) \color{red} - 2ab\color{black} = (a + b)^2 - 2ab$

Now, use a + b = 13 and ab = 25, you will get $a \star b = 119$. Please tell me if you encounter any difficulty in the calculation.