Answer :
The divergence theorem applies since [tex]\mathcal S[/tex] is a closed surface. We have
[tex]\displaystyle\iint_{\mathcal S}\mathbf f(x,y,z)\cdot\mathrm d\mathbf S=\iiint_{\mathcal V}\nabla\times\mathbf f(x,y,z)\,\mathrm dV[/tex]
where [tex]\mathcal V[/tex] is the space enclosed by the surface [tex]\mathcal S[/tex]. The divergence of the given vector field is
[tex]\nabla\times\mathbf f(x,y,z)=\dfrac{\partial(x+3y^5)}{\partial x}+\dfrac{\partial(y+10xz)}{\partial y}+\dfrac{\partial(z-xy)}{\partial z}=1+1+1=3[/tex]
So we can write the volume integral (in spherical coordinates) as
[tex]\displaystyle3\iiint_{\mathcal V}\,\mathrm dV=3\int_{\varphi=0}^{\varphi=\pi/2}\int_{\theta=0}^{\theta=2\pi}\int_{\rho=0}^{\rho=1}\rho^2\sin\varphi\,\mathrm d\rho\,\mathrm d\theta\,\mathrm d\varphi=\frac{4\pi}3[/tex]
[tex]\displaystyle\iint_{\mathcal S}\mathbf f(x,y,z)\cdot\mathrm d\mathbf S=\iiint_{\mathcal V}\nabla\times\mathbf f(x,y,z)\,\mathrm dV[/tex]
where [tex]\mathcal V[/tex] is the space enclosed by the surface [tex]\mathcal S[/tex]. The divergence of the given vector field is
[tex]\nabla\times\mathbf f(x,y,z)=\dfrac{\partial(x+3y^5)}{\partial x}+\dfrac{\partial(y+10xz)}{\partial y}+\dfrac{\partial(z-xy)}{\partial z}=1+1+1=3[/tex]
So we can write the volume integral (in spherical coordinates) as
[tex]\displaystyle3\iiint_{\mathcal V}\,\mathrm dV=3\int_{\varphi=0}^{\varphi=\pi/2}\int_{\theta=0}^{\theta=2\pi}\int_{\rho=0}^{\rho=1}\rho^2\sin\varphi\,\mathrm d\rho\,\mathrm d\theta\,\mathrm d\varphi=\frac{4\pi}3[/tex]