$x=\dfrac{8-m}{m+2}\to {{x}^{2}}=\dfrac{{{m}^{2}}-16m+64}{{{\left( m+2 \right)}^{2}}}$
$y=\dfrac{5}{m+2}\to {{y}^{2}}=\dfrac{25}{{{\left( m+2 \right)}^{2}}}$
$S={{x}^{2}}-{{y}^{2}}$
$S=\dfrac{{{m}^{2}}-16m+64}{{{\left( m+2 \right)}^{2}}}-\dfrac{25}{{{\left( m+2 \right)}^{2}}}$
$S=\dfrac{{{m}^{2}}-16m+39}{{{\left( m+2 \right)}^{2}}}$
$S=\dfrac{{{m}^{2}}+4m+4}{{{\left( m+2 \right)}^{2}}}+\dfrac{-20m-40}{{{\left( m+2 \right)}^{2}}}+\dfrac{75}{{{\left( m+2 \right)}^{2}}}$
$S=\dfrac{{{\left( m+2 \right)}^{2}}}{{{\left( m+2 \right)}^{2}}}-\dfrac{20\left( m+2 \right)}{{{\left( m+2 \right)}^{2}}}+\dfrac{75}{{{\left( m+2 \right)}^{2}}}$
$S=1-\dfrac{20}{m+2}+\dfrac{75}{{{\left( m+2 \right)}^{2}}}$
$S=\dfrac{75}{{{\left( m+2 \right)}^{2}}}-\dfrac{20}{m+2}+1$
$S=\dfrac{75}{{{\left( m+2 \right)}^{2}}}-\dfrac{20}{m+2}+\dfrac{4}{3}-\dfrac{1}{3}$
$S={{\left( \dfrac{\sqrt{75}}{m+2}-\dfrac{2}{\sqrt{3}} \right)}^{2}}-\dfrac{1}{3}\,\,\,\ge \,\,\,-\dfrac{1}{3}$
Dấu “=” xảy ra khi và chỉ khi $\dfrac{\sqrt{75}}{m+2}=\dfrac{2}{\sqrt{3}}\Leftrightarrow m=\dfrac{11}{2}$
Vậy giá trị nhỏ nhất của $S$ bằng $-\dfrac{1}{3}$ khi $m=\dfrac{11}{2}$
