Ta có:
`(a^3-3ab^2)^2+(b^3-3a^2b)^2=5^2+10^2`
`->(a^3)^2-2.a^{3}.3ab^{2}+(3ab^2)^2+(b^3)^2-2.b^{3}.3a^{2}b+(3a^{2}b)^2=125`
`->a^6-6a^{4}b^{2}+9a^{2}b^{4}+b^6-6a^{2}b^{4}+9a^{4}b^{2}=125`
`->a^6+(9a^{4}b^{2}-6a^{4}b^{2})+(9a^{2}b^{4}-6a^{2}b^{4})+b^6=125`
`->a^6+3a^4 b^2 +3a^2 b^4+b^6=125`
`->(a^2)^3+3(a^2)^2 b^2+3a^2 (b^2)^2+(b^2)^3=125`
`->(a^2+b^2)^3=125=5^3`
`->a^2+b^2=5`
Vậy `a^2+b^2=5`