Đáp án:

$S = \left\{ \begin{array}{l}
\left( {\dfrac{{1 + \sqrt {21} }}{2};\dfrac{{1 + \sqrt {21} }}{4}} \right);\left( {\dfrac{{1 - \sqrt {21} }}{2};\dfrac{{1 - \sqrt {21} }}{4}} \right);\left( {\dfrac{{ - 1 + \sqrt {17} }}{2};\dfrac{{ - 1 + \sqrt {17} }}{4}} \right);\left( {\dfrac{{ - 1 - \sqrt {17} }}{2};\dfrac{{ - 1 - \sqrt {17} }}{4}} \right);\\
\left( {2\sqrt 2 ;2\sqrt 2  - 2} \right);\left( { - 2\sqrt 2 ; - 2\sqrt 2  - 2} \right);\left( {\sqrt 2 ;\sqrt 2  - 2} \right);\left( { - \sqrt 2 ; - \sqrt 2  - 2} \right)
\end{array} \right\}$

Giải thích các bước giải:

 Ta có:

$\begin{array}{l}
\left\{ \begin{array}{l}
{x^2} + 2{y^2} - 3xy - 2x + 4y = 0\\
{\left( {{x^2} - 5} \right)^2} = 2x - 2y + 5
\end{array} \right.\\
 \Leftrightarrow \left\{ \begin{array}{l}
\left( {{x^2} - 3xy + 2{y^2}} \right) - 2\left( {x - 2y} \right) = 0\\
{\left( {{x^2} - 5} \right)^2} = 2x - 2y + 5
\end{array} \right.\\
 \Leftrightarrow \left\{ \begin{array}{l}
\left( {x - 2y} \right)\left( {x - y - 2} \right) = 0\\
{\left( {{x^2} - 5} \right)^2} = 2x - 2y + 5
\end{array} \right.\\
 \Leftrightarrow \left\{ \begin{array}{l}
\left[ \begin{array}{l}
x = 2y\\
x - y = 2
\end{array} \right.\\
{\left( {{x^2} - 5} \right)^2} = 2x - 2y + 5
\end{array} \right.\\
 \Leftrightarrow \left[ \begin{array}{l}
\left\{ \begin{array}{l}
x = 2y\\
{\left( {{x^2} - 5} \right)^2} = 2x - 2y + 5
\end{array} \right.\\
\left\{ \begin{array}{l}
x - y = 2\\
{\left( {{x^2} - 5} \right)^2} = 2x - 2y + 5
\end{array} \right.
\end{array} \right.\\
 \Leftrightarrow \left[ \begin{array}{l}
\left\{ \begin{array}{l}
x = 2y\\
{\left( {{x^2} - 5} \right)^2} = x + 5
\end{array} \right.\\
\left\{ \begin{array}{l}
x - y = 2\\
{\left( {{x^2} - 5} \right)^2} = 9
\end{array} \right.
\end{array} \right.\\
 \Leftrightarrow \left[ \begin{array}{l}
\left\{ \begin{array}{l}
x = 2y\\
{x^4} - 10{x^2} - x + 20 = 0
\end{array} \right.\\
\left\{ \begin{array}{l}
x - y = 2\\
\left[ \begin{array}{l}
{x^2} - 5 = 3\\
{x^2} - 5 =  - 3
\end{array} \right.
\end{array} \right.
\end{array} \right. \Leftrightarrow \left[ \begin{array}{l}
\left\{ \begin{array}{l}
x = 2y\\
\left( {{x^2} - x - 5} \right)\left( {{x^2} + x - 4} \right) = 0
\end{array} \right.\\
\left\{ \begin{array}{l}
y = x - 2\\
\left[ \begin{array}{l}
x =  \pm 2\sqrt 2 \\
x =  \pm \sqrt 2 
\end{array} \right.
\end{array} \right.
\end{array} \right.\\
 \Leftrightarrow \left[ \begin{array}{l}
\left\{ \begin{array}{l}
y = \dfrac{x}{2}\\
\left[ \begin{array}{l}
{x^2} - x - 5 = 0\\
{x^2} + x - 4 = 0
\end{array} \right.
\end{array} \right.\\
x = 2\sqrt 2 ;y = 2\sqrt 2  - 2\\
x =  - 2\sqrt 2 ;y =  - 2\sqrt 2  - 2\\
x = \sqrt 2 ;y = \sqrt 2  - 2\\
x =  - \sqrt 2 ;y =  - \sqrt 2  - 2
\end{array} \right. \Leftrightarrow \left[ \begin{array}{l}
x = \dfrac{{1 + \sqrt {21} }}{2};y = \dfrac{{1 + \sqrt {21} }}{4}\\
x = \dfrac{{1 - \sqrt {21} }}{2};y = \dfrac{{1 - \sqrt {21} }}{4}\\
x = \dfrac{{ - 1 + \sqrt {17} }}{2};y = \dfrac{{ - 1 + \sqrt {17} }}{4}\\
x = \dfrac{{ - 1 - \sqrt {17} }}{2};y = \dfrac{{ - 1 - \sqrt {17} }}{4}\\
x = 2\sqrt 2 ;y = 2\sqrt 2  - 2\\
x =  - 2\sqrt 2 ;y =  - 2\sqrt 2  - 2\\
x = \sqrt 2 ;y = \sqrt 2  - 2\\
x =  - \sqrt 2 ;y =  - \sqrt 2  - 2
\end{array} \right.
\end{array}$

Vậy tập nghiệm của phương trình là: 

$S = \left\{ \begin{array}{l}
\left( {\dfrac{{1 + \sqrt {21} }}{2};\dfrac{{1 + \sqrt {21} }}{4}} \right);\left( {\dfrac{{1 - \sqrt {21} }}{2};\dfrac{{1 - \sqrt {21} }}{4}} \right);\left( {\dfrac{{ - 1 + \sqrt {17} }}{2};\dfrac{{ - 1 + \sqrt {17} }}{4}} \right);\left( {\dfrac{{ - 1 - \sqrt {17} }}{2};\dfrac{{ - 1 - \sqrt {17} }}{4}} \right);\\
\left( {2\sqrt 2 ;2\sqrt 2  - 2} \right);\left( { - 2\sqrt 2 ; - 2\sqrt 2  - 2} \right);\left( {\sqrt 2 ;\sqrt 2  - 2} \right);\left( { - \sqrt 2 ; - \sqrt 2  - 2} \right)
\end{array} \right\}$