$\begin{array}{l} 1){\sin ^2}x - \sin 2x + {\cos ^2}x - 1 = 0\\  \Leftrightarrow \left( {{{\sin }^2}x + {{\cos }^2}x} \right) - 1 - \sin 2x = 0\\  \Leftrightarrow 1 - 1 - \sin 2x = 0\\  \Leftrightarrow \sin 2x = 0\\  \Leftrightarrow 2x = k\pi  \Leftrightarrow x = \dfrac{{k\pi }}{2}\left( {k \in \mathbb{Z}} \right)\\ 2){\cos ^2}x - {\sin ^2}x - 2\sin x\cos x = 0\\  \Leftrightarrow \cos 2x - \sin 2x = 0\\  \Leftrightarrow \sqrt 2 \cos \left( {2x + \dfrac{\pi }{4}} \right) = 0\\  \Leftrightarrow \cos \left( {2x + \dfrac{\pi }{4}} \right) = 0\\  \Leftrightarrow 2x + \dfrac{\pi }{4} = \dfrac{\pi }{2} + k\pi \\  \Leftrightarrow 2x = \dfrac{\pi }{4} + k\pi \\  \Leftrightarrow x = \dfrac{\pi }{8} + \dfrac{{k\pi }}{2}\left( {k \in \mathbb{Z}} \right)\\ 3){\sin ^2}x - 3\sin x\cos x = 1\\  \Leftrightarrow {\sin ^2}x - 1 - 3\sin x\cos x = 0\\  \Leftrightarrow {\sin ^2}x - {\sin ^2}x - {\cos ^2}x - 3\sin x\cos x = 0\\  \Leftrightarrow  - {\cos ^2}x - 3\sin x\cos x = 0\\  \Leftrightarrow \cos x\left( {\cos x + 3\sin x} \right) = 0\\  \Leftrightarrow \left[ \begin{array}{l} \cos x = 0\\ \cos x + 3\sin x = 0 \end{array} \right.\\  \Leftrightarrow \left[ \begin{array}{l} x = \dfrac{\pi }{2} + k\pi \\ \sqrt {10} \left( {\dfrac{1}{{\sqrt {10} }}\cos x + \dfrac{3}{{\sqrt {10} }}\sin x} \right) = 0 \end{array} \right.\\  \Leftrightarrow \left[ \begin{array}{l} x = \dfrac{\pi }{2} + k\pi \\ \sqrt {10} \sin \left( {x + \alpha } \right) = 0\left( {\alpha  = \arccos \dfrac{3}{{\sqrt {10} }}} \right) \end{array} \right.\\  \Leftrightarrow \left[ \begin{array}{l} x = \dfrac{\pi }{2} + k\pi \\ x + \alpha  = k\pi  \end{array} \right. \Leftrightarrow \left[ \begin{array}{l} x = \dfrac{\pi }{2} + k\pi \\ x =  - \alpha  + k\pi  \end{array} \right.\left( {k \in \mathbb{Z}} \right) \end{array}$  

Đáp án:

\(\begin{array}{l}
1)\quad S = \left\{k\dfrac{\pi}{2}\ \Bigg|\ k\in\Bbb Z\right\}\\
2)\quad S = \left\{\dfrac{\pi}{8} + k\dfrac{\pi}{2}\ \Bigg|\ k\in\Bbb Z\right\}\\
3)\quad S = \left\{\dfrac{\pi}{2} + k\pi;\ \arctan\left(-\dfrac13\right) + k\pi\ \Bigg|\ k\in\Bbb Z\right\}\end{array}\) 

Giải thích các bước giải:

\(\begin{array}{l}
1)\\
\quad \sin^2x -\sin2x + \cos^2x - 1 =0\\
\Leftrightarrow \sin2x = 0\\
\Leftrightarrow 2x = k\pi\\
\Leftrightarrow x = k\dfrac{\pi}{2}\quad (k\in\Bbb Z)\\
\text{Vậy}\ S = \left\{k\dfrac{\pi}{2}\ \Bigg|\ k\in\Bbb Z\right\}\\
2)\\
\quad \cos^2x - 2\sin x\cos x -\sin^2x = 0\\
\Leftrightarrow \cos2x - \sin2x = 0\\
\Leftrightarrow \sqrt\cos\left(2x + \dfrac{\pi}{4}\right)=0\\
\Leftrightarrow 2x + \dfrac{\pi}{4} = \dfrac{\pi}{2} + k\pi\\
\Leftrightarrow x = \dfrac{\pi}{8} + k\dfrac{\pi}{2}\quad (k\in\Bbb Z)\\
\text{Vậy}\ S = \left\{\dfrac{\pi}{8} + k\dfrac{\pi}{2}\ \Bigg|\ k\in\Bbb Z\right\}
3)\\
\quad \sin^2x -3\sin x\cos x = 1\qquad (2)\\
\bullet\quad \cos x = 0\Leftrightarrow x = \dfrac{\pi}{2} + k\pi\quad (k\in\Bbb Z)\\
(*) \Leftrightarrow \sin^2x = 1\\
\Leftrightarrow \sin x = \pm 1\\
\Leftrightarrow x = \dfrac{\pi}{2} + k\pi\quad (k\in\Bbb Z)\\
\Rightarrow x = \dfrac{\pi}{2} + k\pi\ \text{là một họ nghiệm của $(1)$}\\
\bullet\quad \cos x \ne 0\\
\text{Chia hai vế của phương trình cho $\cos^2x$ ta được:}\\
\quad \tan^2x - 3\tan x = \dfrac{1}{\cos^2x}\\
\Leftrightarrow \tan^2x - 3\tan x = \tan^2 x+  1\\
\Leftrightarrow \tan x = -\dfrac13\\
\Leftrightarrow x = \arctan\left(-\dfrac13\right) + k\pi\quad (k\in\Bbb Z)\\
\text{Vậy}\ S = \left\{\dfrac{\pi}{2} + k\pi;\ \arctan\left(-\dfrac13\right) + k\pi\ \Bigg|\ k\in\Bbb Z\right\}\end{array}\)