Ta có:

Xét \(x=0;y=0\) không là nghiệm của hệ phương trình

Xét \(xe0;ye0\), ta có:\(\left\{{}\begin{matrix}y\left(x^2+1\right)=2x\left(y^2+1\right)\\\left(x^2+y^2\right)\left(1+\dfrac{1}{x^2y^2}\right)=16\end{matrix}\right.\)

\(\Leftrightarrow\left\{{}\begin{matrix}\dfrac{x^2+1}{x}=2.\dfrac{y^2+1}{y}\\x^2+y^2+\dfrac{1}{x^2}+\dfrac{1}{y^2}=16\end{matrix}\right.\)

\(\Leftrightarrow\left\{{}\begin{matrix}x+\dfrac{1}{x}=2\left(y+\dfrac{1}{y}\right)\\\left(x+\dfrac{1}{x}\right)^2+\left(y+\dfrac{1}{y}\right)^2=20\end{matrix}\right.\)

Đặt \(a=x+\dfrac{1}{x};b=y+\dfrac{1}{y}\)

Ta có: \(\left\{{}\begin{matrix}a=2b\\a^2+b^2=20\end{matrix}\right.\)\(\Leftrightarrow\left\{{}\begin{matrix}a=2b\\5b^2=20\end{matrix}\right.\)\(\Leftrightarrow\left\{{}\begin{matrix}a=4\\b=2\end{matrix}\right.\)hoặc\(\left\{{}\begin{matrix}a=-4\\b=-2\end{matrix}\right.\)

*\(\left\{{}\begin{matrix}a=4\\b=2\end{matrix}\right.\)\(\Rightarrow\left\{{}\begin{matrix}x+\dfrac{1}{x}=4\\y+\dfrac{1}{y}=2\end{matrix}\right.\)\(\Leftrightarrow\left\{{}\begin{matrix}x^2-4x+1=0\\y^2-2y+1=0\end{matrix}\right.\)\(\Leftrightarrow\left[{}\begin{matrix}\left\{{}\begin{matrix}x=2-\sqrt{3}\\y=1\end{matrix}\right.\\\left\{{}\begin{matrix}x=2+\sqrt{3}\\y=1\end{matrix}\right.\end{matrix}\right.\)

*\(\left\{{}\begin{matrix}a=-4\\b=-2\end{matrix}\right.\)\(\Rightarrow\left\{{}\begin{matrix}x+\dfrac{1}{x}=-4\\y+\dfrac{1}{y}=-2\end{matrix}\right.\)\(\Leftrightarrow\left\{{}\begin{matrix}x^2+4x+1=0\\y^2+2y+1=0\end{matrix}\right.\)\(\Leftrightarrow\left[{}\begin{matrix}\left\{{}\begin{matrix}x=-2+\sqrt{3}\\y=-1\end{matrix}\right.\\\left\{{}\begin{matrix}x=-2-\sqrt{3}\\y=-1\end{matrix}\right.\end{matrix}\right.\)

Vậy tập nghiệm của hệ phương trình là \(\left(2-\sqrt{3};1\right),\left(2+\sqrt{3};1\right),\left(-2+\sqrt{3};-1\right),\left(-2-\sqrt{3};-1\right)\)