Đáp án:
$\begin{array}{l}
a)Khi:m = 3\\
\Rightarrow \left\{ \begin{array}{l}
x + 3y = 9\\
3x - y = 7
\end{array} \right.\\
\Rightarrow \left\{ \begin{array}{l}
3x + 9y = 27\\
3x - y = 7
\end{array} \right.\\
\Rightarrow \left\{ \begin{array}{l}
10y = 20\\
x + 3y = 9
\end{array} \right.\\
\Rightarrow \left\{ \begin{array}{l}
y = 2\\
x = 9 - 3y = 3
\end{array} \right.\\
Vay\,\left( {x;y} \right) = \left( {3;2} \right)\,khi:m = 3\\
b)\left\{ \begin{array}{l}
x + my = 3m\\
mx - y = {m^2} - 2
\end{array} \right.\\
\Rightarrow \left\{ \begin{array}{l}
mx + {m^2}y = 3{m^2}\\
mx - y = {m^2} - 2
\end{array} \right.\\
\Rightarrow \left\{ \begin{array}{l}
{m^2}y + y = 2{m^2} + 2\\
x + my = 3m
\end{array} \right.\\
\Rightarrow \left\{ \begin{array}{l}
\left( {{m^2} + 1} \right)y = 2\left( {{m^2} + 1} \right)\\
x = 3m - m.y
\end{array} \right.\\
\Rightarrow \left\{ \begin{array}{l}
y = 2\\
x = 3m - 2m = m
\end{array} \right.\\
\Rightarrow \left\{ \begin{array}{l}
x = m\\
y = 2
\end{array} \right.\\
{x^2} - 2x - y > 0\\
\Rightarrow {m^2} - 2.m - 2 > 0\\
\Rightarrow {m^2} - 2m + 1 - 3 > 0\\
\Rightarrow {\left( {m - 1} \right)^2} > 3\\
\Rightarrow \left[ \begin{array}{l}
m - 1 > \sqrt 3 \\
m - 1 < - \sqrt 3
\end{array} \right.\\
\Rightarrow \left[ \begin{array}{l}
m > 1 + \sqrt 3 \\
m < 1 - \sqrt 3
\end{array} \right.
\end{array}$
