+ Ta có bất đẳng thức \(\left | \vec{u}.\vec{v} \right |\leq \left | \vec{u} \right |.\left | \vec{v} \right |\) đẳng thức xảy ra \(\Leftrightarrow \left | cos(\vec{u}\vec{v}) \right |=1\Leftrightarrow \vec{u}\vec{v}\) cùng phương
+ Áp dụng bất đẳng thức trên cho 2 vector \(\vec{u}=(a;\sqrt{2a};1),\vec{v}=(1;\sqrt{2b};c)\) ta được:
\((a+2\sqrt{ab}+c)^2=(\left | a.1+\sqrt{2a}.\sqrt{2b}+1.c \right |)^2\leq \left ( \sqrt{a^2+(\sqrt{2a})^2}+1^2 \right )^2.\left ( \sqrt{1^2+(\sqrt{2b})^2}+c^2 \right )^2\)\(\Rightarrow (a+2\sqrt{ab}+c)^2\leq (a+1)^2(1+2b+c^2)\)
\(\Rightarrow \left ( \frac{a+2\sqrt{ab}+c}{a+1} \right )^2\leq 1+2b+c^2(1)\)
+ Tương tự có \(\left ( \frac{b+2\sqrt{bc}+a}{b+1} \right )^2\leq 1+2c+a^2 \ (2); \left ( \frac{c+2\sqrt{ca}+b}{c+1} \right )^2\leq 1+2a+b^2(3)\)
+ Cộng theo vế (1),(2),(3) ta được
\(P\leq 3+2(a+b+c)+a^2+b^2+c^2=6+2(a+b+c)\leq\) \(6+2\sqrt{3}.\sqrt{a^2+b^2+c^2}=6+2\sqrt{3}.\sqrt{3}=12 \ (4)\)

+ Đẳng thức ở (1) xảy ra \(\Leftrightarrow \frac{a}{1}=\frac{\sqrt{2a}}{2b}=\frac{1}{c}\Leftrightarrow \frac{a^2}{1}=\frac{a}{b}=\frac{1}{c^2}\Leftrightarrow \frac{a}{1}=\frac{1}{b}\wedge \frac{a}{1}=\frac{1}{c}\Leftrightarrow \frac{a}{1}=\frac{1}{b}=\frac{1}{c}\)

+ Tương tự ở (2), (3) nên đẳng thức (4):
\(P=12\Leftrightarrow \left\{\begin{matrix} \frac{a}{1}=\frac{1}{b}=\frac{1}{c}\wedge \frac{b}{1}=\frac{1}{c}=\frac{1}{a}\wedge \frac{c}{1}=\frac{1}{a}=\frac{1}{b}\\ \\ a> 0\wedge b> 0\wedge c> 0\wedge a^2+b^2+c^2=3 \end{matrix}\right.\)
\(\Leftrightarrow \left\{\begin{matrix} b=c\wedge ab=1;c=a\wedge bc=1;a=b\wedge ca=1\\ a> 0;b> 0;a^2+b^2+c^2=3 \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \end{matrix}\right.\)
\(\Leftrightarrow \left\{\begin{matrix} a=b=c>0 \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \\ c^2=b^2=c^2=1\Leftrightarrow a=b=c=1\\ a^2+b^2+c^2=3 \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \end{matrix}\right.\)
Vậy \(MaxP=12\Leftrightarrow a=b=c=1\)