a) Điều kiện xác định $1\le x\le 7$
$\begin{array}{l} x + 2\sqrt {7 - x} = 2\sqrt {x - 1} + \sqrt { - {x^2} + 8x - 7} + 1\\ \Leftrightarrow x + 2\sqrt {7 - x} = 2\sqrt {x - 1} + \sqrt {\left( {7 - x} \right)\left( {x - 1} \right)} + 1\\ \Leftrightarrow x - 1 - 2\sqrt {x - 1} + 2\sqrt {7 - x} - \sqrt {\left( {7 - x} \right)\left( {x - 1} \right)} = 0\\ \Leftrightarrow \sqrt {x - 1} \left( {\sqrt {x - 1} - 2} \right) - \sqrt {7 - x} \left( {\sqrt {x - 1} - 2} \right) = 0\\ \Leftrightarrow \left( {\sqrt {x - 1} - 2} \right)\left( {\sqrt {x - 1} - \sqrt {7 - x} } \right) = 0\\ \Leftrightarrow \left[ \begin{array}{l} \sqrt {x - 1} = 2\\ \sqrt {x - 1} = \sqrt {7 - x} \end{array} \right. \Leftrightarrow \left[ \begin{array}{l} x = 5\\ x = 4 \end{array} \right.(TM)\\ \Rightarrow S = \left\{ {5;4} \right\}\\ b)\\ \left\{ \begin{array}{l} {x^3}\left( {x - y} \right) + {x^2}{y^2} = 1\\ {x^2}\left( {xy + 3} \right) - 3xy = 3 \end{array} \right.\\ \Leftrightarrow \left\{ \begin{array}{l} {x^4} - {x^3}y + {x^2}{y^2} = 1\\ {x^3}y + 3{x^2} - 3xy = 3 \end{array} \right.\\ \Leftrightarrow \left\{ \begin{array}{l} {\left( {{x^2} - xy} \right)^2} + {x^3}y = 1\\ 3\left( {{x^2} - xy} \right) + {x^3}y = 3 \end{array} \right.\\ \Leftrightarrow \left\{ \begin{array}{l} {a^2} + b = 1\left( 1 \right)\\ 3a + b = 3\left( 2 \right) \end{array} \right.\left( {a = {x^2} - xy,b = {x^3}y} \right)\\ \left( 2 \right) \to \left( 1 \right):{a^2} + 3 - 3a = 1\\ \Leftrightarrow {a^2} - 3a + 2 = 0\\ \Leftrightarrow \left[ \begin{array}{l} a = 1 \Rightarrow b = 0\\ a = 2 \Rightarrow b = - 3 \end{array} \right.\\ + \left\{ \begin{array}{l} a = 1\\ b = 0 \end{array} \right. \Rightarrow \left\{ \begin{array}{l} {x^2} - xy = 1\\ {x^3}y = 0 \end{array} \right. \Rightarrow \left\{ \begin{array}{l} y = 0\\ x = 1 \end{array} \right.\left( {y = 0(L)} \right)\\ + \left\{ \begin{array}{l} a = 2\\ b = - 3 \end{array} \right. \Rightarrow \left\{ \begin{array}{l} {x^2} - xy = 2\\ {x^3}y = - 3 \end{array} \right. \Rightarrow \left\{ \begin{array}{l} xy = {x^2} - 2\\ {x^3}y = - 3 \end{array} \right.\\ \Rightarrow {x^3}y = - 3 \Leftrightarrow \left( {{x^2} - 2} \right){x^2} = - 3\\ \Leftrightarrow {x^4} - 2{x^2} + 3 = 0\left( {PTVN} \right)\\ \Rightarrow \left( {x;y} \right) = \left( {1;0} \right) \end{array}$
