The values of $\theta ,\lambda $ for which the following equations,
$\begin{align}
  & \sin \theta x-\cos \theta y+\left( \lambda +1 \right)z=0 \\
 & \cos \theta x+\sin \theta y-\lambda z=0 \\
 & \lambda x+\left( \lambda +1 \right)y+\cos z=0 \\
\end{align}$
Have non trivial solution is,
A. $\theta =n\pi ,\lambda \in R-\left\{ 0 \right\}$
B. $\theta =2n\pi ,\lambda $ is any rational number
C. $\theta =\left( 2n+1 \right)\pi ,\lambda \in R,n\in I$
D. $2\left( n+1 \right)\dfrac{\pi }{2},\lambda \in R,n\in I$

Hint: To solve the above question, we will make use of the concept that if the system of equations in which the determinant of the coefficient is equal to zero, then the system of equations has a non-trivial solution. After forming the determinant, we can expand it, equate it to zero and then we will get the value for $\cos \theta $. Then we can use the general formula for $\cos \theta =\cos \alpha $ as $\theta =2n\pi \pm \alpha $.

Complete step-by-step answer:
We have been asked to find the values of $\theta ,\lambda $ for which the following equations, have non-trivial solution,
$\begin{align}
  & \sin \theta x-\cos \theta y+\left( \lambda +1 \right)z=0 \\
 & \cos \theta x+\sin \theta y-\lambda z=0 \\
 & \lambda x+\left( \lambda +1 \right)y+\cos z=0 \\
\end{align}$
Since, we know that the system of equation in which the determinant of the coefficient is equal to zero, then the system of equation has non-trivial solution. So, we will form a determinant and enter all the coefficients of x, y and the constant term as shown below.
$\left| \begin{align}
  & \sin \theta \text{ }-\cos \theta \text{ }\lambda +1 \\
 & \cos \theta \text{ }\sin \theta \text{ }-\lambda \\
 & \text{ }\lambda \text{ }\lambda +1\text{ }\cos \theta \\
\end{align} \right|=0$
Now, on expanding the determinants, we get the terms as,
$\begin{align}
  & \sin \theta \left[ \sin \theta \cos \theta -\left( -\lambda \right)\left( \lambda +1 \right) \right]+\cos \theta \left( {{\cos }^{2}}\theta +{{\lambda }^{2}} \right)+\left( \lambda +1 \right)\left[ \left( \lambda +1 \right)\cos \theta -\lambda \sin \theta \right]=0 \\
 & \Rightarrow {{\sin }^{2}}\theta \cos \theta +\lambda \left( \lambda +1 \right)\sin \theta +{{\cos }^{3}}\theta +{{\lambda }^{2}}\cos \theta +{{\left( \lambda +1 \right)}^{2}}\cos \theta -\lambda \left( \lambda +1 \right)\sin \theta =0 \\
\end{align}$
We will substitute ${{\sin }^{2}}\theta =1-{{\cos }^{2}}\theta $ in the above expression. So, we get,
$\left( 1-{{\cos }^{2}}\theta \right)\cos \theta +\lambda \left( \lambda +1 \right)\sin \theta +{{\cos }^{3}}\theta +{{\lambda }^{2}}\cos \theta +{{\left( \lambda +1 \right)}^{2}}\cos \theta -\lambda \left( \lambda +1 \right)\sin \theta =0$
By rearranging and cancelling the similar terms, we get,
$\left( 1-{{\cos }^{2}}\theta \right)\cos \theta +{{\cos }^{3}}\theta +{{\lambda }^{2}}\cos \theta +{{\left( \lambda +1 \right)}^{2}}\cos \theta =0$
Now, we will take $\cos \theta $ common form the above expression. So, we will get,
$\begin{align}
  & \cos \theta \left[ 1-{{\cos }^{2}}\theta +{{\cos }^{2}}\theta +{{\lambda }^{2}}+{{\left( \lambda +1 \right)}^{2}} \right]=0 \\
 & \Rightarrow \cos \theta \left[ 1+{{\lambda }^{2}}+{{\left( \lambda +1 \right)}^{2}} \right]=0 \\
\end{align}$
Since, $1+{{\lambda }^{2}}+{{\left( \lambda +1 \right)}^{2}}$ cannot be equal to zero, that means, $\cos \theta =0$. We know that the general solution for $\cos \theta =\cos \alpha $ is given by,
$\begin{align}
  & \theta =2n\pi \pm \alpha \\
 & \Rightarrow \cos \theta =\cos \dfrac{\pi }{2} \\
 & \Rightarrow \theta =2n\pi \pm \dfrac{\pi }{2} \\
 & or\text{ }\theta =\left( 2n+1 \right)\dfrac{\pi }{2} \\
\end{align}$
Hence, $\lambda $ can take any values, so, $\lambda \in R$ and $\theta =\left( 2n+1 \right)\dfrac{\pi }{2}$, where $n\in I$ (integer).
Therefore, the correct option is option D.

Note: The students must be careful while doing the calculations, as there are a lot of calculations involved in this question and should be careful with the signs too in each step. The students might stop after finding $\cos \theta =0$ and may directly write the answer as $\theta =90$. But this is not correct, they must think of the general solution as there can be many values for $\theta $. Also, they should mention that $\lambda $ is an element of R.