An angle between the lines whose direction cosines are given by the equations, L+3m+5n=0 and 5lm-2mn+6nl=0, is
(a) \[{{\cos }^{-1}}\left( \dfrac{1}{8} \right)\]
(b) \[{{\cos }^{-1}}\left( \dfrac{1}{6} \right)\]
(c) \[{{\cos }^{-1}}\left( \dfrac{1}{3} \right)\]
(d) \[{{\cos }^{-1}}\left( \dfrac{1}{4} \right)\]

Hint: In this equation, we have equations of two lines, we need to find the direction vectors of the lines with the help of direction ratios. First we need to simplify using the equations and find the values of (l,m,n) for both lines and use the formula of angle between the lines. \[\cos \theta =\dfrac{\overrightarrow{p}.\overrightarrow{q}}{\left| \overrightarrow{p} \right|\left| \overrightarrow{q} \right|}\]

Complete step by step answer: We have been given two lines, l+3m+5n=0 and line 5lm-2mn+6nl=0. Let us use these two equations and simplify further to find the angle between these two given lines.
We have,
l+3m+5n=0
l=-3m-5n
l= -(3m+5n) -------(1)
We also have,
5lm-2mn+6nl=0
Let us take ‘l’ common and move the rest of the expression on the right hand side keeping the left hand side only ‘l’.
\[\begin{align}
  & \therefore l\left( 5m+6n \right)-2mn=0 \\
 & l\left( 5m+6n \right)=2mn \\
 & l=\dfrac{2mn}{(5m+6n)}---(2) \\
\end{align}\]
From the equation (1) and (2), we get,
\[-(3m+5n)=\dfrac{2mn}{(5m+6n)}\]
After cross multiplication, we get,
\[\begin{align}
  & -\left( 3m+5n \right)\left( 5m+6n \right)=2mn \\
 & \left( 3m+5n \right)\left( 5m+6n \right)+2mn=0 \\
\end{align}\]
Let us expand the expression and get in the form of quadratic equations, we get,
\[\begin{align}
  & 3m\left( 5m+6n \right)+5n\left( 5m+6n \right)+2mn=0 \\
 & 15{{m}^{2}}+18mn+25mn+30{{n}^{2}}+2mn=0 \\
 & 15{{m}^{2}}+45mn+30{{n}^{2}}=0 \\
\end{align}\]
Now, let us divide by ‘15’ throughout the equation, we get
\[{{m}^{2}}+3mn+2{{n}^{2}}=0\]
We have got a quadratic equation; let us further simplify to find the roots.
\[\begin{align}
  & {{m}^{2}}+2mn+mn+2{{n}^{2}}=0 \\
 & {{m}^{2}}+mn+2mn+2{{n}^{2}}=0 \\
 & m\left( m+n \right)+2n\left( m+n \right)=0 \\
 & \left( m+n \right)\left( m+2n \right)=0 \\
\end{align}\]
Therefore, the values we got are m=-n and m=-2n. Now, let us find the direction ratios which will give us the direction vectors of the lines.
For, m=-n;
Let us substitute, m=-n in equation (1)
\[\begin{align}
  & l=-\left[ 3\left( -n \right)+5n \right] \\
 & =-\left[ -3n+5n \right] \\
 & =-[2n] \\
 & l=-2n \\
\end{align}\]
Therefore, we have, l=-2n, m=-n, n=n.
We get, (l, m, n)= (-2n, -n, n)
Direction ratio= (-2, -1, 1)
Hence, we get the direction vector as \[\left( -2 \right)\widehat{i}+\left( -1 \right)\widehat{j}+\left( 1 \right)\widehat{k}\]
Let the direction vector be
\[\overrightarrow{p}=-2\widehat{i}-\widehat{j}+\widehat{k}\]

Therefore, \[\overrightarrow{p}.\overrightarrow{q}={{p}_{x}}{{q}_{x}}+{{p}_{y}}{{q}_{y}}+{{p}_{z}}{{q}_{z}}\]
For m=-2n
Similarly, let us substitute the value m=-2n in equation (1), we get;
\[\begin{align}
  & l=-\left[ 3\left( -2n \right)+5n \right] \\
 & =-\left[ -6n+5n \right] \\
 & =-[-n] \\
 & l=n \\
\end{align}\]
Therefore, we have, l=n, m=-2n, n=n.
(l, m, n) = (n, -2n, n)
Direction ration= (1, -2, 1)
We get the direction vector as \[\left( 1 \right)\widehat{i}+\left( -2 \right)\widehat{j}+\left( 1 \right)\widehat{k}\]
Let us consider the direction vector as \[\overrightarrow{q}=\widehat{i}-2\widehat{j}+\widehat{k}\]
Therefore, \[{{q}_{x}}=1,{{q}_{y}}=-2,{{q}_{z}}=1\]
We know,
Angle between two lines is the angle between direction vectors of the lines that is direction vector \[\overrightarrow{p}\] and \[\overrightarrow{q}\],
\[\cos \theta =\dfrac{\overrightarrow{p}.\overrightarrow{q}}{\left| \overrightarrow{p} \right|\left| \overrightarrow{q} \right|}\]
Also, \[\cos \theta =\dfrac{{{p}_{x}}{{q}_{x}}+{{p}_{y}}{{q}_{y}}+{{p}_{z}}{{q}_{z}}}{\sqrt{p{{x}^{2}}+p{{y}^{2}}+p{{z}^{2}}}.\sqrt{q{{x}^{2}}+q{{y}^{2}}+q{{z}^{2}}}}\]
Let us substitute the values and find the cosine angle between the two lines.
\[\begin{align}
  & \cos \theta =\dfrac{\left( -2 \right)\left( 1 \right)+\left( -1 \right)\left( -2 \right)+\left( 1 \right)\left( 1 \right)}{\sqrt{{{\left( -2 \right)}^{2}}+{{\left( -1 \right)}^{2}}+{{\left( 1 \right)}^{2}}}.\sqrt{{{\left( 1 \right)}^{2}}+{{\left( -2 \right)}^{2}}+{{\left( 1 \right)}^{2}}}} \\
 & =\dfrac{-2+2+1}{\sqrt{4+1+1}.\sqrt{1+4+1}} \\
 & =\dfrac{1}{\sqrt{6}\sqrt{6}} \\
 & \cos \theta =\dfrac{1}{6} \\
 & \theta ={{\cos }^{-1}}\left( \dfrac{1}{6} \right) \\
\end{align}\]


Hence, the angle between the two lines is equal to \[{{\cos }^{-1}}\left( \dfrac{1}{6} \right)\]

Note: Here, the value of ‘m’ could be substituted in either of the equations which are equation (1) or equation (2). If two lines are perpendicular to each other than their direction vectors are also perpendicular. Hence, \[\overrightarrow{p}.\overrightarrow{q}={{p}_{x}}{{q}_{x}}+{{p}_{y}}{{q}_{y}}+{{p}_{z}}{{q}_{z}}=0\]