$\left \{ {{x^3(x-y)+x^2y^2=1} \atop {x^2(xy+3)-3xy=3}} \right.$
⇔ $\left \{ {{x^4-x^2.xy+(xy)^2=1} \atop {x^2(xy+3)-3xy=3}} \right.$
Đặt x² = a; xy = b (với a ≥ 0)
hpt ⇔ $\left \{ {{a^2-ab+b^2=1} \atop {a(b+3)-3b=3}} \right.$
⇔ $\left \{ {{a^2-ab+b^2=1} \atop {ab+3a-3b-3=0}} \right.$
⇔ $\left \{ {{(a-b)^2-ab-1=0(1)} \atop {3(a-b)+b-3=0(2)}} \right.$
(2) ⇔ a-b=$\frac{-ab+3}{3}$
(1) ⇔ $(\frac{-ab+3}{3})^{2}$ - ab - 1 = 0 (*)
Đặt ab = c, pt (*) trở thành
($\frac{-1}{3}$c+1)² - c - 1 = 0
⇔ $\frac{1}{9}$c² - $\frac{2}{3}$c + 1 - c - 1 = 0
⇔ $\frac{1}{9}$c² - $\frac{5}{3}$c = 0
⇔ c($\frac{1}{9}$c - $\frac{5}{3}$) = 0
TH 1: c = 0 ⇔ ab = 0 ⇔ a - b = 1
⇔ $\left \{ {{x^2.x.y=0} \atop {x^2-xy=1}} \right.$
⇔ $\left \{ {{x^3y=0} \atop {x^2-xy=1}} \right.$
TH1: x = 0 ⇒ hpt vô nghiệm
TH2: y = 0 ⇒ x² = 1 ⇒ \(\left[ \begin{array}{l}x=1\\x=-1\end{array} \right.\)
TH 2: $\frac{1}{9}$c - $\frac{5}{3}$ = 0 ⇒ c = 15 ⇒ ab = 15; a - b = -4
⇒ $\left \{ {{x^3y=15} \atop {x^2-xy=-4}} \right.$
⇔ $\left \{ {{xy=15/x^2} \atop {x^2-\frac{15}{x^2}=-4 (**)}} \right.$
(**) ⇔ x^4 - 15 = -4x²
⇔ x^4 + 4x^2 - 15 = 0
⇔ x^4 + 4x^2 + 4 - 19 = 0
⇔ (x^2 + 2)^2 = 19
⇔ x^2 + 2 = $\sqrt{19}$
⇔ x^2 = $\sqrt{19}$ - 2
⇔ x^2 = $\sqrt{\sqrt{19}-2}$
⇒ y = $\frac{x^2+4}{x}$ = $\frac{\sqrt{19}+2}{\sqrt{\sqrt{19}+2}}$
Vậy (x;y) ∈ {(1;0);(-1;0);($\sqrt{19}$ - 2;$\sqrt{\sqrt{19}-2}$)}
