a) $(x+1)^2-(x-1)^2-3.(x+1)(x-1)$
$=x^2+2x+1-x^2+2x-1-3(x^2-1)$
$=(x^2-x^2)+(2x+2x)+(1-1)-3x^2+3$
$=4x-3x^2+3$
b) $5(x+2)(x-2)-\dfrac{1}{2}.(6-8x)^2+17$
$=5.(x^2-4)-\dfrac{1}{2}.(36-96x+64x^2)+17$
$=5x^2-20-18+48x-32x^2+17$
$=(5x^2-32x^2)+(-20-18+17)+48x$
$=-27x^2+48x-21$