Question

Find a unit vector perpendicular to both $\vec a$ and $\vec b$, where $\vec a = \hat i - 2\hat j + 3\hat k$ and $\vec b = \hat i + 2\hat j - \hat k$.

Hint: In cross product (or vector product) of two nonzero vectors $\vec a$ and $\vec b$, the resultant vector is perpendicular to both vectors $\vec a$ and $\vec b$.
So you got the hint, to find a vector perpendicular to two nonzero vectors $\vec a$ and $\vec b$, we have to find the cross product of those two vectors.
Remember that the resultant vector may or may not be a unit vector.
Unit vectors are those vectors whose magnitude is 1.
Therefore, find the unit vector by dividing by vector by its magnitude.

Step 1: Find the cross product of $\vec a$ and $\vec b$.
$\hat a \times \hat b$ is the determinant of the matrix $\left[ {\begin{array}{*{20}{c}} {\hat i}&{\hat j}&{\hat k} \\ 1&{ - 2}&3 \\ 1&2&{ - 1} \end{array}} \right]$
$\hat a \times \hat b$ $= \left| {\begin{array}{*{20}{c}} {\hat i}&{\hat j}&{\hat k} \\ 1&{ - 2}&3 \\ 1&2&{ - 1} \end{array}} \right|$
$\Rightarrow \hat i\left( {2 - 6} \right) - \hat j\left( { - 1 - 3} \right) + \hat k\left( {2 + 2} \right) \\ \Rightarrow \hat i\left( { - 4} \right) - \hat j\left( { - 4} \right) + \hat k\left( 4 \right) \\ \Rightarrow - 4\hat i + 4\hat j + 4\hat k \\$
Let $\vec c = \hat a \times \hat b$
Step 2: Find the unit vector $\hat c$:
$\vec c = - 4\hat i + 4\hat j + 4\hat k$
Magnitude of $\vec c$:
$\left| {\vec c} \right| = \sqrt {{{\left( { - 4} \right)}^2} + {{\left( 4 \right)}^2} + {{\left( 4 \right)}^2}} \\ \Rightarrow {\text{ }} = \sqrt {16 + 16 + 16} \\ \Rightarrow {\text{ }} = 4\sqrt 3 \\$
Unit vector $\hat c = \dfrac{{\vec c}}{{\left| {\vec c} \right|}}$
Therefore, $\hat c$ $= \dfrac{{ - 4\hat i + 4\hat j + 4\hat k}}{{4\sqrt 3 }}$
So, $\hat c$$= - \dfrac{1}{{\sqrt 3 }}\hat i + \dfrac{1}{{\sqrt 3 }}\hat j + \dfrac{1}{{\sqrt 3 }}\hat k$
Final answer: Unit vector perpendicular to both $\vec a$ and $\vec b$, where $\vec a = \hat i - 2\hat j + 3\hat k$ and $\vec b = \hat i + 2\hat j - \hat k$ is $- \dfrac{1}{{\sqrt 3 }}\hat i + \dfrac{1}{{\sqrt 3 }}\hat j + \dfrac{1}{{\sqrt 3 }}\hat k$.

The unit vector $\hat i$ is along the direction of the x-axis, the unit vector $\hat j$ is along the direction of the y-axis, and the unit vector $\hat k$ is along the direction of the z-axis. Thus, $\hat i,{\text{ }}\hat j{\text{ }}\& {\text{ }}\hat k$ are unit vectors mutually perpendicular to each other.
Note: You can check the final answer by finding the magnitude of the vector $\hat c$.
$\left| {\hat c} \right| =$$\sqrt {{{\left( { - \dfrac{1}{{\sqrt 3 }}} \right)}^2} + {{\left( {\dfrac{1}{{\sqrt 3 }}} \right)}^2} + {{\left( {\dfrac{1}{{\sqrt 3 }}} \right)}^2}}$
$\Rightarrow {\text{ }} = \sqrt {\dfrac{1}{3} + \dfrac{1}{3} + \dfrac{1}{3}} \\ \Rightarrow {\text{ }} = \sqrt {\dfrac{3}{3}} = \sqrt 1 \\ \Rightarrow {\text{ }} = 1 \\$
$\hat a \times \hat b = - \left( {\hat b \times \hat a} \right)$. So, if you have calculated $\hat b \times \hat a$$= \left| {\begin{array}{*{20}{c}} {\hat i}&{\hat j}&{\hat k} \\ 1&2&{ - 1} \\ 1&{ - 2}&3 \end{array}} \right|$, the resultant vector i.e. $4\hat i - 4\hat j - 4\hat k$ still be a vector perpendicular to both $\vec a$ and $\vec b$but in the direction opposite to $\hat a \times \hat b$.
Also, the cross product of two nonzero vectors $\vec a$ and $\vec b$ is the product of the magnitude of both vectors $\vec a$ and $\vec b$, and sine of the angle between them. i.e.
$\hat a \times \hat b = \left| {\hat a} \right|\left| {\hat b} \right|\sin \theta {\text{ }}\hat n$, where $\theta$ is the acute angle between vectors $\vec a$ and $\vec b$. Here $\hat n$ is the unit vector perpendicular to the plane containing vectors $\vec a$ and $\vec b$.