The dot product of a vector with vectors \[\hat{i}+\hat{j}-3\hat{k},\hat{i}+3\hat{j}-2\hat{k},2\hat{i}+\hat{j}+4\hat{k}\] are 0,5 and 8 respectively. Find the vector.

Hint: In this problem, we must solve the given dot product of vectors with vectors and find the vector. As we can see that given dot products with vectors can be converted into simplest form and considered as equations 1, 2 and 3.so, we can subtract different equations to find out the required vector.

Complete step-by-step solution:
Let us assume required vector be \[a\hat{i}+b\hat{j}+c\hat{k}\]
Given that dot product of a vector with vectors \[\hat{i}+\hat{j}-3\hat{k},\hat{i}+3\hat{j}-2\hat{k},2\hat{i}+\hat{j}+4\hat{k}\] are 0,5 and 8 respectively.
Dot product with vector \[\hat{i}+\hat{j}-3\hat{k}=a+b-3c=0..........\left( 1 \right)\]
Dot product with vector \[\hat{i}+3\hat{j}-2\hat{k}=a+3b-2c=5........\left( 2 \right)\]
 Dot product with vector \[2\hat{i}+\hat{j}+4\hat{k}=2a+b+4c=8...........\left( 3 \right)\]
Subtracting equation (2) with equation (1)
$\Rightarrow a+3b-2c-\left(a+b-3c \right) = 5-0$
$\Rightarrow a+3b-2c-a-b+3c = 5$
$\Rightarrow 2b+c=5..........................\left( 4 \right)$
Subtracting equation (3) with 2\[\times \] [equation (2)]
\[\begin{align}
  & \Rightarrow 2a+b+4c-2\left( a+3b-2c \right)=8-2\left( 5 \right) \\
 & \Rightarrow 2a+b+4c-2a-6b+4c=8-10 \\
 & \Rightarrow -5b+8c=-2.......\left( 5 \right) \\
\end{align}\]
Subtracting equation (5) with 8\[\times \] [equation (2)]
\[\begin{align}
  & \Rightarrow -5b+8c-8\left( 2b+c \right)=-2-8\left( 5 \right) \\
 & \Rightarrow -5b+8c-16b-8c=-2-40 \\
 & \Rightarrow -21b=-42..........\left( 5 \right) \\
\end{align}\]
From equation (5), we get b value as
\[\therefore b=\left( \dfrac{-42}{-21} \right)=2\]
Putting the value of b i.e., \[b=2\] into equation (4)
 We can write the equation (4) as
\[c=5-2b\]
Substituting the value of ‘b’ in above equation, then we get the ‘c’ value as
\[\begin{align}
  & c=5-2\times 2=5-4 \\
 & \therefore c=1 \\
\end{align}\]
So, we have \[b=2\] and \[c=1\], to find the value of a, we have to substitute the b and c values in equation (1)
\[a+b-3c=0\]
\[\begin{align}
  & \Rightarrow a+2-3\times 1=0 \\
 & \Rightarrow a+2-3=0 \\
 & \therefore a-1=0 \\
 & a=1 \\
\end{align}\]
Now, we have found out the a, b and c values
Substituting the \[a=1,b=2\]and \[c=1\]in required vector
\[\Rightarrow a\hat{i}+b\hat{j}+c\hat{k}\]
\[\therefore \hat{i}+2\hat{j}+\hat{k}\]
Therefore, the required vector is \[\hat{i}+2\hat{j}+\hat{k}\].

Note: It can also be explained in another form i.e., given dot product of vectors is converted to required form and solved using matrix method. Finding out the determinants with given equations and solving them to find out the required vector.
 We have required vector as
\[\Rightarrow a\hat{i}+b\hat{j}+c\hat{k}\]
Given dot product of a vectors with vectors are
\[a+b-3c=0..........\left( 1 \right)\]
\[a+3b-2c=5........\left( 2 \right)\]
\[2a+b+4c=8...........\left( 3 \right)\]
Writing the equations (1), (2) and (3) into matrix form
i.e.,\[\left[ \begin{matrix}
   1 & 1 & -3 \\
   1 & 3 & -2 \\
   2 & 1 & 4 \\
\end{matrix} \right]\]
finding the determinant of above matrix
\[\Delta =\left| \left[ \begin{matrix}
   1 & 1 & -3 \\
   1 & 3 & -2 \\
   2 & 1 & 4 \\
\end{matrix} \right] \right|\]
On solving, we get
\[\therefore \Delta =21\]
Now replacing first column with (0,5,8)
\[{{\Delta }_{1}}=\left| \left[ \begin{matrix}
   0 & 1 & -3 \\
   5 & 3 & -2 \\
   8 & 1 & 4 \\
\end{matrix} \right] \right|=21\]
 Now replacing second column with (0,5,8)
\[{{\Delta }_{2}}=\left| \left[ \begin{matrix}
   1 & 0 & -3 \\
   1 & 5 & -2 \\
   2 & 8 & 4 \\
\end{matrix} \right] \right|=42\]
Now replacing third column with (0,5,8)
\[{{\Delta }_{3}}=\left| \left[ \begin{matrix}
   1 & 1 & 0 \\
   1 & 3 & 5 \\
   2 & 1 & 8 \\
\end{matrix} \right] \right|=21\]
For finding the values of a, b and c,
\[\begin{align}
  & a=\dfrac{{{\Delta }_{1}}}{\Delta }=\dfrac{21}{21}=1 \\
 & b=\dfrac{{{\Delta }_{2}}}{\Delta }=\dfrac{42}{21}=2 \\
 & c=\dfrac{{{\Delta }_{3}}}{\Delta }=21=1 \\
\end{align}\]
Where a=1, b=2 and c=3
Then the required vector is \[\hat{i}+2\hat{j}+\hat{k}\].