Giải thích các bước giải:

$DK:x,y \ge 1$

Ta có:

$\begin{array}{l}
\left\{ \begin{array}{l}
\sqrt {x - 1}  + \sqrt {y - 1}  = 3\\
xy + x + y = {x^2} - 2{y^2}
\end{array} \right.\\
 \Leftrightarrow \left\{ \begin{array}{l}
\sqrt {x - 1}  + \sqrt {y - 1}  = 3\\
{x^2} - xy - 2{y^2} - \left( {x + y} \right) = 0
\end{array} \right.\\
 \Leftrightarrow \left\{ \begin{array}{l}
\sqrt {x - 1}  + \sqrt {y - 1}  = 3\\
\left( {x + y} \right)\left( {x - 2y} \right) - \left( {x + y} \right) = 0
\end{array} \right.\\
 \Leftrightarrow \left\{ \begin{array}{l}
\sqrt {x - 1}  + \sqrt {y - 1}  = 3\\
\left( {x + y} \right)\left( {x - 2y - 1} \right) = 0
\end{array} \right.\\
 \Leftrightarrow \left\{ \begin{array}{l}
\sqrt {x - 1}  + \sqrt {y - 1}  = 3\\
\left[ \begin{array}{l}
x + y = 0\\
x - 2y - 1 = 0
\end{array} \right.
\end{array} \right.\\
 \Leftrightarrow \left[ \begin{array}{l}
\left\{ \begin{array}{l}
\sqrt {x - 1}  + \sqrt {y - 1}  = 3\\
x =  - y
\end{array} \right.\left( I \right)\\
\left\{ \begin{array}{l}
\sqrt {x - 1}  + \sqrt {y - 1}  = 3\\
x = 2y + 1
\end{array} \right.\left( {II} \right)
\end{array} \right.
\end{array}$

+) Giải $\left( I \right) \Leftrightarrow \left\{ \begin{array}{l}
x - 1 + y - 1 + 2\sqrt {\left( {x - 1} \right)\left( {y - 1} \right)}  = 3\\
x =  - y
\end{array} \right.$

$\begin{array}{l}
 \Leftrightarrow \left\{ \begin{array}{l}
2\sqrt {\left( { - y - 1} \right)\left( {y - 1} \right)}  = 5\\
x =  - y
\end{array} \right.\\
 \Leftrightarrow \left\{ \begin{array}{l}
\sqrt {1 - {y^2}}  = \dfrac{5}{2}\left( {vn,do:1 - {y^2} \le 1,\forall y \Rightarrow \sqrt {1 - {y^2}}  \le 1 < \dfrac{5}{2}} \right)\\
x =  - y
\end{array} \right.
\end{array}$

$\to (I)$ vô nghiệm 

+) Giải $\left( {II} \right) \Leftrightarrow \left\{ \begin{array}{l}
\sqrt {x - 1}  + \sqrt {y - 1}  = 3\\
x = 2y + 1
\end{array} \right.$

$\begin{array}{l}
 \Leftrightarrow \left\{ \begin{array}{l}
\sqrt {2y}  + \sqrt {y - 1}  = 3\\
x = 2y + 1
\end{array} \right.\\
 \Leftrightarrow \left\{ \begin{array}{l}
2y + y - 1 + 2\sqrt {2y\left( {y - 1} \right)}  = 9\\
x = 2y + 1
\end{array} \right.\\
 \Leftrightarrow \left\{ \begin{array}{l}
2\sqrt {2y\left( {y - 1} \right)}  = 10 - 3y\\
x = 2y + 1
\end{array} \right.\\
 \Leftrightarrow \left\{ \begin{array}{l}
10 - 3y \ge 0\\
4.2y\left( {y - 1} \right) = {\left( {10 - 3y} \right)^2}\\
x = 2y + 1
\end{array} \right.\\
 \Leftrightarrow \left\{ \begin{array}{l}
y \le \dfrac{{10}}{3}\\
{y^2} - 52y + 100 = 0\\
x = 2y + 1
\end{array} \right.\\
 \Leftrightarrow \left\{ \begin{array}{l}
y \le \dfrac{{10}}{3}\\
\left( {y - 50} \right)\left( {y - 2} \right) = 0\\
x = 2y + 1
\end{array} \right.\\
 \Leftrightarrow \left\{ \begin{array}{l}
y \le \dfrac{{10}}{3}\\
\left[ \begin{array}{l}
y = 50\left( l \right)\\
y = 2\left( c \right)
\end{array} \right.\\
x = 2y + 1
\end{array} \right.\\
 \Leftrightarrow \left\{ \begin{array}{l}
y = 2\\
x = 5
\end{array} \right.\left( {tm} \right)
\end{array}$

Vậy hệ có nghiệm duy nhất $(x;y)=(5;2)$